[RFC] A Possible Next Step for GGUF Model Adapters

[Feijia1231]

Summary

PR #97 introduces the new GGUFModelFiles and GGUFLoadPlan based adapter
interface. PR #98 builds on that refactor to add Qwen3.5/3.6 multimodal GGUF
and MTP support, with dedicated adapters for dense, MoE, vision, and MTP
models.

While looking at the changes in these two PRs, I was wondering whether the
adapter design could be made a little easier to extend as more architectures
are added.

A longer-term goal is to make it practical to support a broader range of
GGUF models published by Unsloth, including dense, MoE, multimodal, and
diffusion models:

https://huggingface.co/models?library=gguf&sort=trending&search=unsloth

Motivation

The current refactor makes the loading lifecycle clearer, but model support
still depends on architecture-specific behavior inside adapters.

The model families being added now already require different combinations
of:

• QKV splitting or merging;
• merged-column splitting;
• MoE expert expansion;
• multimodal projector handling;
• MTP-specific fallback behavior;
• tensor-name transformations.

As more GGUF models are added, these operations may become increasingly
difficult to share between adapters. Diffusion loading also continues to
use a separate integration path.

Possible Direction

One possible direction would be to keep the GGUFModelFiles and
GGUFLoadPlan interfaces introduced by PR #97, and gradually move some of
the model-specific behavior toward reusable transformations:

GGUFModelFiles
    → GGUFLoadPlan
    → name mapping
    → tensor transformations
    → model.load_weights()

For example, QKV splitting, MoE expert expansion, and multimodal projector
mapping could potentially be shared across adapters instead of being
implemented separately each time.

The existing Transformers-based mapping can remain as a fallback. Explicit
architecture mappings can be added incrementally where they make model
support more predictable and reduce dependence on dummy Transformers models.

It may also be useful for diffusion loading to share the same file, plan, and
transformation concepts in the future, without changing its current
integration boundary all at once.

Possible Next Step

If this direction seems useful after PR #97 and PR #98, a first small step
could be:

1. Look for transformations shared by the current dense, MoE, vision, and
   MTP adapters.
2. Extract one or two of them into small reusable components.
3. Try the components with another model family.
4. Add regression tests for the relevant transformations, including
   tensor-parallel and sharded GGUF loading.

Ideally, adding another model family would mostly involve a localized name
mapping and a small number of model-specific transformations, rather than
another large branch in the generic adapter. If this direction makes sense,
I would be happy to help work on the first step.

References

• PR #97: Refactor WeightsAdapter design
• PR #98: Support Qwen3.5/3.6 multimodal GGUF and MTP

[lee-b]

*Q2*/*Q3* would be very much welcome. This seems to be unsupported so far, right?

[Isotr0py]

For example, QKV splitting, MoE expert expansion, and multimodal projector
mapping could potentially be shared across adapters instead of being
implemented separately each time.

Actually, imo, this can be quite difficult to be unified and shared across most of models. If you look into llama.cpp's conversion script, you will find that it almost implements corresponding converter for each model arch:
https://github.com/ggml-org/llama.cpp/tree/master/conversion

The motivation to abstract modular WeightsAdapter is the mess in previous Transformers-based model converter, we have added massive patches there during previous model support, which has become a disaster of maintenance:

vllm-gguf-plugin/vllm_gguf_plugin/weights_adapter/default.py

Lines 60 to 142 in d358f56

	if model_type in ("deepseek_v3", "deepseek_v2"):
	model_type = "deepseek2"
	for idx in range(config.num_hidden_layers):
	gguf_to_hf_name_map[f"blk.{idx}.exp_probs_b.bias"] = (
	f"model.layers.{idx}.mlp.gate.e_score_correction_bias"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_down_exps.weight"] = (
	f"model.layers.{idx}.mlp.experts.0.down_proj.weight"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_gate_exps.weight"] = (
	f"model.layers.{idx}.mlp.experts.0.gate_proj.weight"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_up_exps.weight"] = (
	f"model.layers.{idx}.mlp.experts.0.up_proj.weight"
	)
	sideload_params.append(
	regex.compile(
	f"model\\.layers\\.{idx}"
	r"\.mlp\.experts\.[0-9]+\.(gate|up|down)_proj\.weight"
	)
	)
	if model_type in ("qwen2_moe", "qwen3_moe"):
	model_type = model_type.replace("_", "")
	for idx in range(config.num_hidden_layers):
	gguf_to_hf_name_map[f"blk.{idx}.ffn_down_exps.weight"] = (
	f"model.layers.{idx}.mlp.experts.0.down_proj.weight"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_gate_exps.weight"] = (
	f"model.layers.{idx}.mlp.experts.0.gate_proj.weight"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_up_exps.weight"] = (
	f"model.layers.{idx}.mlp.experts.0.up_proj.weight"
	)
	sideload_params.append(
	regex.compile(
	f"model\\.layers\\.{idx}"
	r"\.mlp\.experts\.[0-9]+\.(gate|up|down)_proj\.weight"
	)
	)
	if model_type == "olmoe":
	for idx in range(config.num_hidden_layers):
	gguf_to_hf_name_map[f"blk.{idx}.ffn_down_exps.weight"] = (
	f"model.layers.{idx}.mlp.experts.0.down_proj.weight"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_gate_exps.weight"] = (
	f"model.layers.{idx}.mlp.experts.0.gate_proj.weight"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_up_exps.weight"] = (
	f"model.layers.{idx}.mlp.experts.0.up_proj.weight"
	)
	sideload_params.extend(
	[
	regex.compile(
	f"model\\.layers\\.{idx}"
	r"\.mlp\.experts\.[0-9]+\.(gate|up|down)_proj\.weight"
	),
	regex.compile(
	f"model\\.layers\\.{idx}"
	r"\.mlp\.experts\.(gate_up_proj|down_proj)"
	),
	]
	)
	if model_type == "minimax_m2":
	model_type = "minimax-m2"
	for idx in range(config.num_hidden_layers):
	gguf_to_hf_name_map[f"blk.{idx}.exp_probs_b.bias"] = (
	f"model.layers.{idx}.block_sparse_moe.e_score_correction_bias"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_down_exps.weight"] = (
	f"model.layers.{idx}.block_sparse_moe.experts.0.w2.weight"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_gate_exps.weight"] = (
	f"model.layers.{idx}.block_sparse_moe.experts.0.w1.weight"
	)
	gguf_to_hf_name_map[f"blk.{idx}.ffn_up_exps.weight"] = (
	f"model.layers.{idx}.block_sparse_moe.experts.0.w3.weight"
	)
	sideload_params.append(
	regex.compile(
	f"model\\.layers\\.{idx}"
	r"\.block_sparse_moe\.experts\.(gate_up_proj|down_proj)"
	)
	)

I will recommend move these models' patches (deepseek, olmoe, minimax...) out to become a standalone weights adapter first, then it will be much clear that what should be shared and what is model-specific.

[Isotr0py]

BTW, I think another pain point in current plugin design is the config parser, because we need to extract and recover hf_config from GGUF file, which might be unavoidable to need different workaround for each model arch:
https://github.com/vllm-project/vllm-gguf-plugin/blob/main/vllm_gguf_plugin/config_parser.py

[Feijia1231]

BTW, I think another pain point in current plugin design is the config parser, because we need to extract and recover hf_config from GGUF file, which might be unavoidable to need different workaround for each model arch: https://github.com/vllm-project/vllm-gguf-plugin/blob/main/vllm_gguf_plugin/config_parser.py

Hi,
Thanks for the detailed feedback, which was really useful. I agree that the DeepSeek, OLMoE, and MiniMax patches are already quite architecture-specific. Trying to unify them upfront would probably be premature.

I checked how vLLM and SGLang handle OLMoE. Their architecture-specific weight-loading logic stays with the OLMoE model implementation, which seems like a useful reference for separating the model-specific part in this plugin. As a small experiment, I moved the OLMoE-specific mapping and sideload rules into a standalone adapter, while keeping the generic GGUF loading flow in the default adapter. The adapter regression tests and related plugin tests are passing locally.

My current understanding is that this kind of separation would be a reasonable first step, then we can see what is genuinely shared across models? The config parser seems better treated as a separate follow-up for now？

[Isotr0py]

My current understanding is that this kind of separation would be a reasonable first step, then we can see what is genuinely shared across models? The config parser seems better treated as a separate follow-up for now？

Yes, and config parser is more related to multimodal because of extra vision config, which is not too critical at least for now.