[Models] Support the Muse Glimmer dflash draft model in GGUF

[WhatGhost]

Support the Muse Glimmer dflash draft model in GGUF

Stacked on #113 — the first commit belongs to that PR; the dflash change is the second commit 7e9ecd4 alone. I'll rebase onto main once #113 lands.

Summary

#113 left GGUF speculative decoding blocked before any adapter is consulted. This unblocks it and adds the Muse Glimmer dflash draft, so the Q4_K_XL backbone can be served with its 2.56B assistant head drafting for it.

The draft is the opposite of the backbone: none of the four conversions #113 undoes applies here. Its Q/K rows are already in NEOX order, its norms carry no folded offset, and its Q/K norms are learned rather than synthesized. Reusing the backbone's rules would rewrite correct weights and nothing would say so — the target verifies every token, so the output stays fluent while the draft's proposals quietly stop being accepted. The name map is therefore the whole conversion, and several tests assert that a transformation did not happen.

The three blockers

A draft has no hf_config_path. The target gets one for free: create_model_config rewrites model to the config source and keeps the file in model_weights. A draft's .gguf path reaches ModelConfig intact, so its config is looked for in the file's own directory — which, for a draft shipped next to the target it drafts for, holds the target's config. speculative_config now accepts hf_config_path, and a directory without a config.json fails with an explanation instead of Unrecognized model. The redirect is idempotent: create_speculative_config runs twice, and without remembering the original path the second pass would keep the config directory as the weights source and silently load the unquantized checkpoint sitting there.

EAGLEConfig hides the model type. A dflash draft is wrapped on its way into the engine, reporting model_type == "eagle" with the real config on .model, and the wrap happens after the config parser runs. So architecture() sees the bare type while the loader sees the wrapped one, and an adapter matching only on the bare type stops matching exactly when the weights are about to be mapped — the fallback adapter then fails on an architecture it has never heard of.

The fused KV buffer needs dense weights. The head fuses every layer's KV projection at the end of loading and reads qkv_proj.weight, which a quantized layer does not have. Only Q/K/V are unpacked, about a sixth of the draft, so it still loads at roughly a third of its unquantized size. Building the buffer from packed bytes instead is not an option anyway: K is Q4_K and V is Q6_K, so their rows differ in width.

Supporting changes

Declarations reach a draft through the config dict. A target's layers are built against the very GGUFConfig the loader extends; a draft's are not, since it rebuilds its own from hf_config.quantization_config. Everything the loader records on the shared object therefore reaches the target and never the draft. The loader now writes the declaration into that dict as well, and GGUFConfig.from_config reads it back.

Exemptions can be declared by suffix. unquantized_modules cannot express "wherever this occurs": a fused layer matches by asking whether a declared name contains the layer's full runtime path, so every declaration spells out a prefix and a layer index. Those are knowable for a target and not for a draft, whose layers vLLM numbers after the target's — the draft's five appear as model.layers.62..66. dense_module_suffixes drops that coupling.

ReplicatedLinear gets a GGUF loader. It is the one linear layer with no weight_loader_v2, and its v1 loader asserts the parameter already has the loaded weight's shape. GGUF parameters start empty and take their shape from the packed bytes, so the draft's fc layer tripped that assertion on its first tensor.

Testing

46 synthetic tests in test_muse_glimmer_dflash_gguf.py, none needing a checkpoint. They pin the name map as a bijection, the match through the EAGLE wrapper, the Q/K/V unpacking and the packed bytes everything else keeps, and — as negative controls — that the norms, the final norm, the Q/K norms and the Q/K row order all come through untouched. The rest covers the two structural differences that each produced a silent-wrong-weights bug while this was written: a declaration surviving the rebuild from the config dict, and a suffix declaration surviving the layer renumbering.

End-to-end against the real 30B, GGUF target with GGUF draft, 3 speculative tokens: 60.8% acceptance at a mean accepted length of 2.82, against a BF16 baseline of 51.0–53.2% at 2.53 over the same prompts. Per position, 83.5/60.4/38.5% against 72.5/47.1/33.3%.

Also checked against the real weights outside CI: the name map is an exact 58↔58 bijection onto the assistant checkpoint, and the tensors are a plain requantization of it — no row permutation, no folded norm offset, and Q/K norms that differ from what the backbone's scale factor would synthesize.

Known limitations

A draft's quantization config is resolved by a path GGUF cannot take. get_quant_config reads hf_config.quantization_config and falls back to hf_overrides; a GGUF file has neither, because vLLM always hands a draft a callable hf_overrides so config transforms applied to the target reach it too, and the fallback rejects anything that is not a dict. That rejection is the whole obstacle, and it looks like an oversight: the two keys read from hf_overrides there can only exist on a dict, so a callable means "neither key is present" rather than something being wrong, and raising makes the file-based lookup below it unreachable for every draft. This PR plants a {"quant_method": "gguf"} marker to select the first branch instead; it comes out once that check stops raising upstream.

Q/K/V are dequantized rather than fused while packed. Keeping them packed needs _build_fused_kv_buffers to accept quantized layers, which would cost the fused-GEMM fast path for every model that uses it.