[GGUF] Add config fallback and Gemma4/Qwen3.5 mappings

[lesj0610]

Purpose

This PR adds generic GGUF config resolution and architecture-specific tensor mappings for Gemma4 and Qwen3.5, so that GGUF quantizations of these models can be loaded without manual --hf-config-path workarounds.

Most GGUF repos (e.g., Unsloth quantizations) ship only quantized weights — no config.json, no sidecar tokenizer files. Previously, remote GGUF repos without config.json hit a hard ValueError. This PR resolves that by falling back to the original HF base model referenced in the GGUF repo's model card or local GGUF metadata for config loading, while keeping the default tokenizer on the GGUF file so Transformers can read the embedded GGUF tokenizer/vocab.

This PR has three logical sections that can be reviewed independently.

Section 1: Generic GGUF config resolver and embedded tokenizer handling

Files: gguf_utils.py, config.py, model.py, arg_utils.py, registry.py

Adds resolve_gguf_config_source() which resolves where a GGUF model should load its HF config from. The fallback chain is:

1. GGUF repo / local parent directory (if it has config.json)
2. Verified HF base model from model card base_model (remote) or GGUF metadata general.base_model.*.repo_url (local)
3. Transformers GGUF metadata parser (for architectures Transformers supports natively)
4. Fail with a clear error

Both get_config() and maybe_override_with_speculators() now use the same fallback chain for local and remote GGUF. The previous hard ValueError for remote GGUF without config.json is removed.

For tokenizer loading, the default tokenizer remains the GGUF model path/repo. This preserves the existing Transformers gguf_file path and lets Transformers read the embedded GGUF tokenizer/vocab. Users can still pass an explicit --tokenizer when they intentionally want a sidecar or base-model tokenizer.

MTP / speculative decoding models that ship as separate GGUF files (e.g., Qwen3.6-35B-A3B-MTP) also go through maybe_override_with_speculators() for config detection. The hf_config_path is now forwarded to that function so that explicit user overrides are respected, and the same base model fallback applies when the MTP GGUF repo lacks config.json.

When the resolver falls back to a base model, the GGUF repo's revision is not forwarded to the base model lookup, since GGUF repo revisions have no meaning for the original HF model.

Implicit GGUF metadata redirects also do not inherit trust_remote_code. A warning is logged when the fallback is used; users who intentionally trust remote code from the base repository can pass an explicit --hf-config-path.

Section 2: Gemma4 GGUF architecture adapter

Files: gguf_loader.py, gemma4.py, gemma4_mm.py

Adds Gemma4-specific GGUF tensor name mappings that gguf-py's get_tensor_name_map() does not yet cover:

• Vision tower: v.blk.N.* → model.vision_tower.encoder.layers.N.* (13 tensor types per layer)
• Vision embeddings: v.std_bias, v.std_scale, v.patch_embd.weight, v.position_embd.weight
• MM projector: mm.input_projection.weight → model.embed_vision.embedding_projection.weight
• MoE experts: ffn_gate_up_exps.weight, ffn_down_exps.weight → fused expert loader targets
• Router/layer scalars: ffn_gate_inp.scale, ffn_down_exps.scale, layer_output_scale.weight

Gemma4 patch embedder weight (GGUF 4D Conv2D → vLLM 2D Linear) is handled via a model-level weight_loader attribute on vision_tower.patch_embedder.input_proj.weight, following the same pattern as Qwen3 VL's patch embed loader. No loader-level model-specific transforms.

Gemma4 fused MoE qweight_type for GGUF is routed to the correct expert parameter in gemma4.py.

Section 3: Qwen3.5 GGUF adapter

Files: gguf_loader.py, qwen3_5.py, qwen3_vl.py

• Adds qwen3_5_moe to the GGUF architecture alias map (qwen3_5_moe → qwen35moe).
• Fixes MoE expert tensor prefix for multimodal Qwen3.5: model.language_model.layers instead of model.layers.
• Adds ssm_dt.bias → linear_attn.dt_bias mapping for Qwen3.5 GDN linear attention layers.
• Adds Qwen3.5 VL vision merger mappings (mm.0/2.weight/bias).
• Adds a generic GGUF tuple shard splitter in qwen3_5.py for MergedColumnParallelLinear layers (e.g., in_proj_qkvz) that receive fused GGUF tensors with tuple shard_id. The splitter respects packed_factor and falls back to the existing weight_loader if conditions are not met.
• Adds a patch embed weight_loader for Qwen3 VL that handles GGUF 4D Conv2D → HF 5D Conv3D with temporal dimension expansion.

Test Plan

pytest tests/transformers_utils/test_config.py -k gguf -q          # config resolver
pytest tests/transformers_utils/test_utils.py -k gguf -q           # gguf_utils
pytest tests/tokenizers_/test_registry.py -k gguf -q               # tokenizer resolver
pytest tests/models/test_gguf_download.py -q                       # loader/mapping/transform
ruff check / ruff format --check on all changed files

Test Result

Local GGUF config resolution verified for:

• Qwen3.6-35B-A3B GGUF (local, no config.json in GGUF, base_model metadata present)
• Qwen3.6-35B-A3B MTP GGUF (local, same pattern)
• Gemma4-26B-A4B-it GGUF (local, config.json present in GGUF repo)
• Gemma4-26B-A4B-it-qat GGUF (local, config.json present in GGUF repo)

Notes:

• The Gemma4 vision tower mappings will become redundant once gguf-py adds native Gemma4 support. They are isolated in _add_gemma4_gguf_mappings() for easy removal.
• Full end-to-end multimodal serving (image input) and MTP speculative serving with GGUF have not been verified in this PR. The scope is config/tokenizer resolution and weight name mapping.

AI assistance: Codex, Claude, and Gemini.

---

Essential Elements of an Effective PR Description Checklist

• The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
• The test plan, such as providing test command.
• The test results, such as pasting the results comparison before and after, or e2e results
• (Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 839977c464

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[Isotr0py]

We're migrating the GGUF support to OOT plugin (https://github.com/vllm-project/vllm-gguf-plugin), can you open a PR there instead?

[lesj0610]

@Isotr0py I moved GGUF-specific parts to plugin repo as suggested.
PR: vllm-project/vllm-gguf-plugin#21