[pull] master from mudler:master - #1510
Merged
Merged
Conversation
* docs(llama-cpp): clarify multimodal speculative decoding Update the speculative decoding guidance now that modern llama.cpp backends can combine mmproj-based vision with speculative decoding, including MTP. Document compatibility checks, draft acceptance statistics, VRAM tradeoffs, and a combined configuration example. Assisted-by: Codex:GPT-5.6-Sol [gh] [OpenStack] [Docker] Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com> * docs(llama-cpp): clarify multimodal MTP references Distinguish the upstream change that removed the general multimodal speculative restriction from the later change that added MTP with explicit vision compatibility. Assisted-by: Codex:GPT-5.6-Sol [gh] [Docker] Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com> --------- Signed-off-by: Abdullah Mansour <abdullahmansour.marketing@gmail.com>
- middleware.md: the 'default detector' link used #instance-wide-defaults; the heading is 'Instance-wide default detector' - the advanced/reference landing pages linked an ../installation/ directory that does not exist in docs/content; dropped the dead bullets (deployment content lives under getting-started)
- the PR template's 'Signed commits' anchor pointed at a CONTRIBUTING section that does not exist; repointed at the Commit messages section - the longcat-video backend README linked a docs page that was never committed; replaced the dead link with plain text - formal-verification/README.md used ../../../ for five in-repo packages (escaping the repo root); fixed to ../
- api-errors.md documented LOCALAI_SUBTLEKEY_COMPARISON (missing the KEY underscore); the code defines LOCALAI_SUBTLE_KEY_COMPARISON, so the documented variable silently did nothing - cli-reference.md documented a --csrf flag / $LOCALAI_CSRF env that do not exist, with inverted semantics; the actual flag is --disable-csrf (LOCALAI_DISABLE_CSRF), 'Disable CSRF middleware (enabled by default)'
Add the official F16 GGUF build of the compact Apache-2.0 Qwen3-family chat model with its native 2K context limit. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the compact LightOnOCR model to the OCR section with Q8_0 and F16 builds. Each build uses its matching vision projector. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the original-dtype GGUF for LocalAI\x27s audio.cpp backend. The entry exposes multilingual voice cloning and records the model license.\n\nAssisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add a compact NVFP4 build with its embedded MTP head and vision projector. This gives Blackwell-capable llama.cpp hosts a speculative option alongside the existing Q4 and Q8 builds. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the Q4_K_M and Q8_0 GGUF builds for a compact Qwen3.5-based reasoning and tool-use distillation. The variant pair lets LocalAI choose higher fidelity when the host has enough memory while keeping the Q4 build as the installable fallback. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the artifact-backed VoxCPM2 model for multilingual speech synthesis, voice design, and controllable voice cloning. Assisted-by: Codex:gpt-5.6 [Codex] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
DFM Mimir is a new permissively licensed Danish and English HRM-Text model. LocalAI already ships a vLLM version with native support for its architecture. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The new importance-matrix builds reduce Laguna S 2.1 from the existing 96 GB default to 73.9 GB or 54.4 GB. Add both as selectable llama.cpp variants. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4_K_M and Q8_0 llama.cpp builds plus the official F16 GGUF for Mixedbread mxbai-embed-large-v1. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add the text-only Q4_K_M build for private red-team, blue-team, and security operations workloads. Configure the supported Gemma 4 model for llama.cpp with its verified Hugging Face checksum. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Add Q4_K_XL and Q8_K_XL llama.cpp builds with the shared vision projector. Enable the preserved MTP head for speculative decoding. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
LLM-jp 4 provides a recent Japanese and English reasoning model on the supported Llama architecture. Add its official Q4 and BF16 GGUF builds so hosts can select the fidelity that fits. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
importNpmLock turns the same-version hono override into a file: tarball that conflicts with the direct dependency (EOVERRIDE). Pass --legacy-peer-deps so the flake build can proceed without touching package.json (open #11633). Fixes #11804 Signed-off-by: lei_lei <96427312+leilei3167@users.noreply.github.com>
WhisperX silently returned a plain transcript when diarization lacked the Hugging Face token required to load pyannote. Reject that request clearly so callers do not mistake missing speaker labels for a successful diarization. Convert WhisperX seconds to the nanosecond duration unit used by the transcription API. Assisted-by: Codex:gpt-5 Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
The Intel backend installs PyTorch XPU wheels, but Qwen ASR only checked CUDA and MPS. Every Intel model therefore loaded on the CPU. Select XPU when available and place the model on xpu:0. Keep the existing CUDA, MPS, and CPU placement behavior. Assisted-by: Codex:GPT-5 [apply_patch] [gh] Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
…11874) A quantization job that runs with no client on its progress stream stays "queued" forever, in the API and in state.json, while the finished artifact sits on disk. state.json was written once by StartJob, and the only code that advanced a job afterwards lived inside the stream callback of StreamProgress, so job state depended on somebody watching it. The backend's progress stream cannot simply gain a second reader: each job owns one queue.Queue and QuantizationProgress pops from it, so two consumers split the updates rather than both seeing them. The stream has to be opened exactly once per job. StartJob now starts watchProgress on the application context (the request context is done as soon as the handler returns). That goroutine is the single reader: it applies each update to the job -- in the cross-replica store and in state.json, terminal statuses still winning over late updates -- and republishes it in-process. StreamProgress becomes a pure reader over that fan-out and no longer loads a backend or opens a stream. A client attaching to a job that has already finished, including one hydrated from disk after a restart, gets a final event built from the stored job instead of blocking. Two paths used to end a client's stream by breaking the gRPC connection and now release it explicitly: StopJob kills the backend, so it publishes the stopped event itself; and a stream that ends without a terminal update means the backend is gone, so the job is recorded as failed rather than left running forever. Signed-off-by: Tai An <antai12232931@outlook.com>
#11772 exempted Temperature from the zero-filter in both backend adapters, because proto3 has no field presence and an explicit 0 is indistinguishable from "unset". Seed has exactly the same property and is still filtered: if proto_field != "Temperature" and value in (None, 0, 0.0, [], False, ""): continue A caller pinning `"seed": 0` for a reproducible run therefore gets a random seed instead, with no error and no log line — the one case where the failure is invisible precisely because the request looked deliberate. Both adapters now share a named tuple of fields whose zero is meaningful, so the next one is added in one place rather than as a second special case. Deliberately left filtered: top_k, top_p, min_p and the penalties. Their zero is not a value a caller means — sglang disables top_k with -1, not 0, so forwarding 0 there would turn a default into an invalid argument. Verified on the sglang backend (Qwen3.5-MoE, arm64): with the temperature fix alone, two identical requests at temperature 0 are byte-identical, but pinning seed 0 has no effect until this change. Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
vLLM's engine-based reasoning parsers derive their initial state from the
chat template kwargs. Qwen3Parser:
chat_kwargs = kwargs.get("chat_template_kwargs", {}) or {}
self.thinking_enabled = chat_kwargs.get("enable_thinking", True)
Constructed as ReasoningParser(tokenizer) the flag defaults to True, so the
parser starts in the REASONING state. A completion produced with thinking
disabled contains no tags at all, and every reasoning parser shape then
reports the whole answer as reasoning:
- engine-based parsers classify it by initial state;
- BaseThinkingReasoningParser hits its documented "may not generate start
token" fallback and returns (model_output, None).
Either way `content = c if c is not None else generated_text` turns that
into a duplicate: a Qwen3 model answering "391" with thinking off comes back
as reasoning_content="391" AND content="391".
Measured against Qwen3.5-MoE on vLLM 0.28, non-streaming:
before thinking on reasoning=202 content="391"
thinking off reasoning="391" content="391" <- duplicated
after thinking on reasoning=192 content="391"
thinking off reasoning="" content="391"
Forward the kwargs the prompt was rendered with, which is what vLLM's own
OpenAI server does; parsers that do not accept the argument keep the plain
constructor.
_split_reasoning() covers the older parser shape, which has no initial state
to set. It only reclassifies when the parser exposes a start/end token pair
and neither the completion nor the prompt ever opened a reasoning block.
Truncated reasoning (block open, end token never arrived) stays reasoning,
and parsers without that token pair are left untouched.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
…plate (#11621) With `template.use_tokenizer_template: true` the sglang and vllm backends render the prompt themselves via `tokenizer.apply_chat_template()`, and they hand it plain string content. A chat template only emits the model's own media tokens when the content is a list of parts, so the rendered prompt carries no `<|vision_start|><|image_pad|><|vision_end|>`. The pixels do reach the engine (`image_data` / `multi_modal_data`), but both engines locate them by scanning the prompt for that token, so they are discarded silently: HTTP 200, no warning, and the model answers as if no image had been attached. Add `attach_media_parts()` to the shared `python_utils` helper and call it in both backends: the last user turn is rebuilt as `[{"type": "image"} * n, {"type": "video"} * n, {"type": "text", ...}]` before templating, which makes the template emit the placeholders. The pixels keep travelling out of band exactly as before. Text-only requests are untouched - with no media the helper returns None and the original string-content path runs unchanged. If a template cannot iterate content parts (a text-only model), the parts render is caught and the request falls back to the previous string-content prompt instead of failing. Signed-off-by: Tai An <antai12232931@outlook.com>
Qwen3-style chat templates append the opening <think> tag to the *prompt*
when thinking is enabled. The model therefore never generates it and emits
only the reasoning text plus the closing </think>.
sglang's ReasoningParser keys off the opening tag:
in_reasoning = self._in_reasoning or self.think_start_token in text
if not in_reasoning:
return StreamingParseResult(normal_text=text)
so with such a template the entire completion — reasoning and answer, the
raw </think> in between — is returned as content and reasoning_content
stays empty, no matter how reasoning_parser is configured.
sglang's own OpenAI server handles this via
force_reasoning = (self.template_manager.force_reasoning
or self._get_reasoning_from_request(request))
This backend has no template manager, so derive the same signal from the
rendered prompt: if it ends with the detector's think_start_token, the tag
was prefilled and the parser is constructed with force_reasoning=True.
Structured decoding is the exception, and it matters: a grammar applies
from the first token, so the model cannot emit the closing tag even though
the template opened the block. The whole completion is schema output and
belongs in content — forcing there files it as reasoning and returns an
empty answer. Measured against a JSON-schema code audit: 10107 characters
of "reasoning", zero content. sglang's own server keeps the two apart for
the same reason; its grammar backend owns the reasoning prefix when a
reasoning parser is configured.
force_reasoning is only passed when it is meant to be True, so detector
defaults (DeepSeek-R1 already defaults to True) are untouched, and a
prompt without a prefilled tag behaves exactly as before — which matters,
because forcing unconditionally makes an answer generated with thinking
off disappear into reasoning_content.
The construction is factored into _new_reasoning_parser() so the streaming
and non-streaming paths, which previously built the parser separately,
cannot drift apart.
Signed-off-by: pos-ei-don <1822533+pos-ei-don@users.noreply.github.com>
Squashed merge of #11633. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Squashed merge of #11819. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Squashed merge of #11836. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Squashed merge of #11278. Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )