Skip to content

[BugFix] Fix quant file selection in download_gguf - #123

Merged
Isotr0py merged 2 commits into
vllm-project:mainfrom
ricky-chaoju:fix/gguf-quant-file-resolution
Aug 30, 2026
Merged

[BugFix] Fix quant file selection in download_gguf#123
Isotr0py merged 2 commits into
vllm-project:mainfrom
ricky-chaoju:fix/gguf-quant-file-resolution

Conversation

@ricky-chaoju

Copy link
Copy Markdown
Contributor

download_gguf hands its patterns to snapshot_download, which matches them with fnmatch against repo-relative paths where * spans /, then re-matches the same patterns locally with glob, where * does not. A repo that publishes each quant in its own folder therefore downloads fine and then dies with Downloaded GGUF files not found. That is most of the big MoE repos: all 363 gguf files in unsloth/Kimi-K2-Instruct-GGUF sit in subdirectories, and unsloth/DeepSeek-V3.1-GGUF, Qwen/Qwen3-235B-A22B-GGUF, and unsloth/gpt-oss-120b-GGUF are the same. Using ** with recursive=True makes the local scan see what was downloaded.

The same patterns also match the projector. For :BF16, the candidates are mmproj-BF16.gguf and gemma-3-4b-it-BF16.gguf, and the (count("-"), name) tie-break prefers the mmproj, so the vision tower comes back as the backbone. download_mmproj already resolves the projector, so I exclude it here and rank on the file name instead of the full path.

One behaviour change worth flagging: unsloth/gemma-3-4b-it-GGUF:F16 publishes no F16 backbone, only BF16. It used to resolve to the projector and load it as the model; now it raises the existing "not found" error.

test_download_gguf_subdir mocked glob.glob into returning a subdirectory hit for any matching pattern, so it passed against code that could not do it. I left it and added four tests that build a real directory tree.

Signed-off-by: RickyChen / 陳昭儒 <ricky.chen@infinirc.com>
Signed-off-by: RickyChen / 陳昭儒 <ricky.chen@infinirc.com>
@ricky-chaoju

Copy link
Copy Markdown
Contributor Author

@Isotr0py Can you please take a look? Thanks!

@Isotr0py
Isotr0py merged commit eebdc42 into vllm-project:main Aug 30, 2026
2 checks passed
@ricky-chaoju
ricky-chaoju deleted the fix/gguf-quant-file-resolution branch September 3, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants