Currently, GGUF seems to refer to the repo and load config files like config.json.
But in real cases, many GGUF files do not have repo link metadata inside the GGUF file. So when serving GGUF, we need to put the original model's json files and jinja file together. However, when GGUF is distributed on Hugging Face, this kind of data is usually not included.
So I am wondering what is better behavior here.
Should vllm-gguf-plugin just stop because there is no config and maybe OOM happens?
Or is it better to stop earlier and show a clear message, like:
"Config files are missing. Please put config.json, tokenizer config, and chat template jinja files in the same path as the GGUF file, or specify the original model/tokenizer repo."
I think the second way may be better, because OOM message can make users misunderstand that it is only GPU memory problem.
Maybe it would be helpful if the plugin checks missing config files before model loading, and gives a clear error or warning message.
P.S. Since vLLM usually needs external config/tokenizer files for GGUF serving, I think it would be helpful to make this requirement explicit when those files are missing. This can avoid confusion for users who expect single-file GGUF serving behavior.
Currently, GGUF seems to refer to the repo and load config files like
config.json.But in real cases, many GGUF files do not have repo link metadata inside the GGUF file. So when serving GGUF, we need to put the original model's json files and jinja file together. However, when GGUF is distributed on Hugging Face, this kind of data is usually not included.
So I am wondering what is better behavior here.
Should vllm-gguf-plugin just stop because there is no config and maybe OOM happens?
Or is it better to stop earlier and show a clear message, like:
"Config files are missing. Please put
config.json, tokenizer config, and chat template jinja files in the same path as the GGUF file, or specify the original model/tokenizer repo."I think the second way may be better, because OOM message can make users misunderstand that it is only GPU memory problem.
Maybe it would be helpful if the plugin checks missing config files before model loading, and gives a clear error or warning message.
P.S. Since vLLM usually needs external config/tokenizer files for GGUF serving, I think it would be helpful to make this requirement explicit when those files are missing. This can avoid confusion for users who expect single-file GGUF serving behavior.