[pull] master from mudler:master - #1453
Merged
Merged
Conversation
…1409cd1b027fbc9` (#11542) ⬆️ Update ikawrakow/ik_llama.cpp Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
…11541) ⬆️ Update vllm-project/vllm-metal (darwin) Signed-off-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: mudler <2420543+mudler@users.noreply.github.com>
Adds eight curated vllm-cpp entries to the model gallery. Until now the backend had gallery coverage only for MiniMax-H3 video, so serving text on it meant hand-writing engine_args. The flagship tier is what vllm.cpp gates its correctness and speed claims on: Qwen3.6-27B and Qwen3.6-35B-A3B in NVFP4, each with a speculative sibling (MTP on both, DFlash on the 27B). Qwen3-Coder-30B-A3B covers agentic tool use, and Qwen3-4B / Qwen3-0.6B in bf16 are the entries that run where NVFP4 cannot, CPU included. Three details are load-bearing rather than incidental: - The 27B entries pin revision 890bdef7. That repository was later re-quantized in place from NVFP4 to FP8 W8A8 under the same name, so an unpinned entry resolves to different weights and reports nothing. - Qwen3-Coder names tool_parser: qwen3_coder explicitly. Its dialect is byte-identical on the wire to step3p5's, so chat-template sniffing cannot separate them and auto-detection picks wrong. - enable_prefix_caching is deliberately left unset everywhere. It defaults on for dense models and off for the GDN hybrids, and that per-model default is the right answer. num_blocks is sized per model from its real KV footprint rather than copied between entries, which ranges from 20 KiB/token on the 35B to 144 KiB/token on the 4B. Docs: adds features/vllm-cpp.md covering installation, the model table, the pinning rationale and how to choose between the speculative variants, and cross-links it from the existing engine_args reference. It also records that the CUDA images are built for Blackwell only, which is narrower than vllm.cpp's own ten-architecture release and makes an otherwise cryptic "no kernel image is available" failure legible. Verified: gallery suite green; all eight decode and validate as a ModelConfig. qwen3-0.6b-vllm-cpp confirmed end to end on a real cluster, chat plus engine-parsed tool_calls. The NVFP4 entries are not yet runtime-verified: no available node has kernels for them. Assisted-by: Claude Code:claude-opus-5[1m] [Read] [Bash] [Edit] Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )