Hi — thanks for maintaining this plugin, it's the only realistic path to GGUF on vLLM since the in-tree deprecation. Wanted to flag a gap and offer something concrete rather than just a bare request.
What's missing
hy_v3 (Tencent Hy3, HYV3ForCausalLM) isn't in gguf.MODEL_ARCH_NAMES — confirmed by grepping the repo (zero hits for hy_v3/hyv3) and checking open PRs/issues (none reference it either, as of this writing). Loading any hy_v3 GGUF currently fails with Unknown gguf model_type: hy_v3 in the plugin's weight loader, even though HYV3ForCausalLM is natively implemented in vLLM core for safetensors/NVFP4 — the gap is specifically in this plugin's GGUF tensor-name mapping, not in vLLM's model support.
What we already got working
We hit this ourselves and built a working adapter rather than just wait: build_name_map_from_files() — dynamically constructs the tensor→module mapping straight from the GGUF file's own metadata at load time, instead of relying on a static entry in MODEL_ARCH_NAMES. Confirmed on our end:
- CPU-side dry-run gate: 1298/1298 tensors resolved, zero unmapped, zero name collisions, against a
hy3-1M-MTP-*.gguf (satgeze conversion, standard llama.cpp blk.N.* naming, includes the MTP/NextN layer).
- GPU load: weights actually land on-device (66.14 GiB/worker at TP=2), server reaches health, generates coherent output on our test battery.
- Had to separately work around an unrelated FlashInfer JIT issue on SM120/CUDA<12.9 (
VLLM_USE_FLASHINFER_SAMPLER=0) to get past profile_run — not a plugin issue, just adjacent noise worth mentioning in case anyone else hits it testing this on Blackwell.
Honest caveat — this is a starting point, not a finished patch
Model loads and runs, but end-to-end it's not yet a case for using this over llama.cpp or native vLLM: cold TTFT at ~28k context came in at 51.6s (vs sub-second warm from prefix-cache hits, and vs single-digit seconds on other paths we tested), so on-the-fly GGUF dequant at this architecture's scale needs more than just the name-mapping fix to be genuinely competitive — likely dequant-kernel-level work beyond what our adapter touches. Flagging this so nobody reads "loads and runs" as "ready to serve."
Offer
Happy to share the adapter code (dynamic name-map + the small plugin registration shim) if it's useful as a starting point for someone picking this up, and to test any attempt on our hardware (2x RTX PRO 6000 Blackwell, TP=2) — we already have the measurement harness (TTFT/decode benchmarks, a quality battery) set up from getting this far, so turnaround on testing a patch would be quick.
Hi — thanks for maintaining this plugin, it's the only realistic path to GGUF on vLLM since the in-tree deprecation. Wanted to flag a gap and offer something concrete rather than just a bare request.
What's missing
hy_v3(Tencent Hy3,HYV3ForCausalLM) isn't ingguf.MODEL_ARCH_NAMES— confirmed by grepping the repo (zero hits forhy_v3/hyv3) and checking open PRs/issues (none reference it either, as of this writing). Loading any hy_v3 GGUF currently fails withUnknown gguf model_type: hy_v3in the plugin's weight loader, even thoughHYV3ForCausalLMis natively implemented in vLLM core for safetensors/NVFP4 — the gap is specifically in this plugin's GGUF tensor-name mapping, not in vLLM's model support.What we already got working
We hit this ourselves and built a working adapter rather than just wait:
build_name_map_from_files()— dynamically constructs the tensor→module mapping straight from the GGUF file's own metadata at load time, instead of relying on a static entry inMODEL_ARCH_NAMES. Confirmed on our end:hy3-1M-MTP-*.gguf(satgeze conversion, standard llama.cppblk.N.*naming, includes the MTP/NextN layer).VLLM_USE_FLASHINFER_SAMPLER=0) to get pastprofile_run— not a plugin issue, just adjacent noise worth mentioning in case anyone else hits it testing this on Blackwell.Honest caveat — this is a starting point, not a finished patch
Model loads and runs, but end-to-end it's not yet a case for using this over llama.cpp or native vLLM: cold TTFT at ~28k context came in at 51.6s (vs sub-second warm from prefix-cache hits, and vs single-digit seconds on other paths we tested), so on-the-fly GGUF dequant at this architecture's scale needs more than just the name-mapping fix to be genuinely competitive — likely dequant-kernel-level work beyond what our adapter touches. Flagging this so nobody reads "loads and runs" as "ready to serve."
Offer
Happy to share the adapter code (dynamic name-map + the small plugin registration shim) if it's useful as a starting point for someone picking this up, and to test any attempt on our hardware (2x RTX PRO 6000 Blackwell, TP=2) — we already have the measurement harness (TTFT/decode benchmarks, a quality battery) set up from getting this far, so turnaround on testing a patch would be quick.