[NektoDron]

What this changes

Add opt-in native DeepSeek GBNF with request-local state, speculative verification and HTTP preflight. The build option MAYA_DS4_GRAMMAR defaults to OFF.

Answers

• The default answers are unchanged (same tokens as before).
• It changes answers, and it is opt-in (off by default). Enable MAYA_DS4_GRAMMAR=ON and supply an explicit grammar.

Tested on

Ubuntu 26.04.1 LTS, AMD EPYC 9654 96-core CPU, approximately 625 GiB visible RAM, one active NVIDIA RTX 5090 with 32 GB VRAM, CUDA 13.1, GCC 14, SM120. The machine has two RTX 5090 cards. The GPU fixture uses the custom Q3_K / FP8-Engram DeepSeek pack, context 524288, four native request slots, CPU64 and DSpark auto. Source versions and historical qualification scope are recorded below.

Speed (if this is a speed change)

The historical measurements and their scope are retained below. The November 14 qualification covers build and feature behavior.

Checks

• Builds on the platform I tested
• The tests I touched pass
• Docs updated where a setting or behaviour changed

November 14 rebase

Rebased on main 6651b5b (v1.1.63). Current head is be709c1df779eebf39c6f5d86a79c34e4111869f.

Public-head Linux checks passed with 978 server tests (40 skipped), 941 installer/tools tests (40 skipped), all 18 focused grammar tests and the HIP mapping check.

An additional optional Transformers float64 oracle check failed with the local Torch/Transformers stack because grouped matrix multiplication rejects Double input. The same failure was reproduced on unchanged main 6651b5b. The tools suites passed using the dependency set from the existing CI workflow, where this optional check is skipped.

The standalone public head passed CUDA Release builds with the default grammar option OFF and an explicit ON build. The default target graph excludes the grammar library and checker. Metadata/vocabulary grammar-state and path checks passed. Build tools were CUDA 13.1, GCC 14 and SM120.

Fresh integrated RTX 5090 checks at 728cd8c5 passed 24 grammar cases and 20 sleep cases, covering four native request slots, grammar changes with prefix reuse, cancellation and follow-up, busy exclusion, explicit wake and concurrent automatic wake. The fixture combines the current #855 and #856 source with the exact #854 F16 loader required by the custom pack. The standalone public #856 CUDA build was checked separately.

Actual speculative execution was observed in the stochastic grammar probe with draft_n=1 and draft_n_accepted=1. The marker restored with 3615 cached tokens. The observed checkpoint took 14.364 s and explicit wake took 9.225 s. These are feature-smoke observations on this fixture.

Each OFF/ON build accepted 20 legacy-field requests across both APIs and streaming modes. OFF rejected explicit native GBNF with 400, and ON accepted the nullable-sibling grammar request. Comparing the legacy outputs, visible text matched in 19/20 pairs, reasoning text in 12/20, and both together in 11/20. Legacy JSON parsing succeeded for 13/20 OFF responses and 14/20 ON responses. Six OpenAI pairs returned the same invalid {"ok":<|end|>true} text in both builds. One additional OFF OpenAI streaming json_schema response had that marker, while its ON counterpart was valid JSON with a newline. All ten Anthropic final responses were valid JSON, and eight of those pairs had different reasoning text. The route checks establish HTTP/protocol acceptance.

The rebase preserves ablation defaults, mixed-GPU startup, pool request state and remote bias/penalty propagation. Linked pool and mixed-GPU split layouts run sampling in the other engine, so they advertise zero grammar capabilities and reject an explicit grammar before request state changes. The mixed-GPU helper layout keeps its sampling head local and retains grammar support.

Grammar exhaustion and pool-link failure share the native sentinel -2. A grammar request now returns exhaustion directly to the existing DONE grammar path. The source regression checks capability reporting, guard ordering and this fallback decision. Frontend coverage checks rejection before validation or generation.

Earlier implementation and qualification

Add opt-in native DeepSeek GBNF decoding with request-local grammar state and HTTP preflight. Target logits are constrained before the existing sampler. DSpark verification advances a cloned grammar through accepted proposals, and emitted tokens update the request state.

Review fixes

• Native normalization is selected only by an explicit non-null grammar, guided_grammar or structured_outputs.grammar. Existing JSON aliases keep their previous server handling, including structured_outputs, guided_json, json_schema and ordinary response_format. Nullable fields are treated as absent. The three reported rejections reproduce on the prior implementation and pass after the fix.
• MAYA_DS4_GRAMMAR now defaults to OFF for every build. A fresh default DeepSeek build succeeded and its target graph contains neither the grammar library nor the checker. An explicit ON build produced the engine and metadata-only checker and passed native vocabulary/state tests.
• Checker tests construct platform-native temporary paths and executable suffixes. Windows command-path coverage includes spaces and backslashes. Fake checker/engine protocols launch through Python explicitly, including the real four-route/replay/cancellation test, so those tests run on Windows as well as Linux. The fixture source is written as UTF-8.
• A dedicated grammar-windows job in the existing tests workflow runs all 16 grammar adapter/protocol tests on Windows/Python 3.12. Local Linux execution passed all 16. The actual Windows job awaits hosted fork-workflow approval.

Validation

• The earlier qualification used main 00655481 (v1.1.57) and public head 6fb4de7. The rebase retains the new GLM CMake include. The head passed 250 relevant Linux grammar/server/Responses/GLM API tests (2 skipped).
• The default OFF CUDA build and explicit ON build succeeded; the default target graph excludes the grammar library/checker. Real legacy JSON requests were accepted with both builds. Existing native vocabulary, grammar-state, path and request-local tests passed.
• Fresh RTX 5090 qualification used the integrated 73b4221 source based on cf807579, with the F16 loader and sleep changes. The grammar native implementation matches the reviewed PR; this is not a standalone new-base native build.
• In four fresh processes (OFF/ON/ON/OFF), all measured prompts had 1,135–1,136 new tokens with no prefix reuse, and 130 output tokens. All six restrictive-GBNF outputs matched the required 64-digit JSON array. DSpark offered 700 drafts and accepted 578.
• With DS4_PATH_PICK=0 in the private benchmark only, median prefill/decode rates were OFF/no grammar 583.1/126.7, ON/no grammar 592.2/111.7, ON/grammar 580.0/90.1 tok/s. The pooled nongrammar decode difference is confounded by intervening grammar requests and different speculative histories.
• The clean first-request control, before any grammar request in each fresh process, gave median decode 107.75 → 107.95 tok/s, with byte-identical outputs and identical draft counts across all four processes. This is a small single-prompt control, not proof of zero overhead generally. Other mixed-history outputs differed, and an earlier same-build A/A control also varied; universal greedy identity is not claimed.
• The current grammar path copies target logits to host memory for masking and verifies grammar-constrained speculative rows sequentially. The constrained workload has a measurable decode cost. Statistical sampling-distribution equivalence remains unmeasured.
• Windows/Python 3.12 grammar protocol coverage is committed, but actual hosted execution awaits approval. Windows and HIP GPU hardware were unavailable; the maintainer has offered testing.

Full hardware/model, per-process and pooled rates, source scope, exploratory failures and limits: #856 (comment)

Follow-up request-order control: four fresh OFF/ON/ON/OFF processes executed identical six-request unconstrained sequences; two ON processes repeated the mixed sequence with grammar requests in the same positions. P0/P1 matched across all six. P2 varied even between two OFF runs with identical order and GBNF compiled out, as well as within ON. At that November 9 request-order stage, equal ordering still allowed variation and its cause was unresolved. All 36 generation requests returned 200; all six constrained outputs satisfied the grammar.

Later on November 9, matched native traces located the earlier unconstrained-answer variation after adaptive expert placement. The first residency difference selected CPU versus GPU execution for an expert, followed by a small hidden-state difference that amplified before text diverged. Freezing placement removed variation in that workload. This diagnosis applies to that earlier matched reproduction. Detailed placement/logit investigation

[mw00]

Thanks @NektoDron, a big and well-tested piece of work. Grammar support is a new feature, so it's not in this release; we'll decide on it later. If it goes ahead we'd need three things: requests that work today shouldn't start getting 400s (right now structured_outputs, guided_json and json_schema are refused on every model); the grammar build should be opt-in, so a normal DeepSeek build doesn't compile all of llama.cpp; and the tests should also pass on Windows (one path check assumes /).

[NektoDron]

Addressed the three points.

• Native validation now activates only for explicit non-null GBNF fields. Existing structured JSON aliases retain their earlier handling, including nullable fields. I reproduced the three reported rejections on the previous implementation and added request/HTTP regressions. Real DeepSeek accepted the legacy JSON requests with the grammar feature both disabled and enabled.
• Grammar compilation defaults to OFF. Both fresh default and explicit-ON CUDA builds passed, and the default target graph contains neither the grammar library nor its checker.
• Replaced the Unix-specific path expectation with native temporary paths and executable suffixes, added a Windows command-path case, and made the fake native/checker subprocess tests portable. All 16 grammar tests pass locally on Linux. A Windows/Python 3.12 job is committed in the tests workflow. Its actual Windows result still awaits fork-workflow approval.

The integrated candidate also passed the full Linux server/tools suites and 24/24 real-GPU grammar checks. The PR body separates those results from the pending Windows run. Understood about the feature hold.

[mw00]

Thanks, build-off by default and request-local state is the right shape. Please share: (1) speed with a grammar on vs off (prefill and decode separately, GPU/CPU/RAM, model+quant), (2) that requests without a grammar give identical output and speed, and (3) the result with DSpark verification on a real run. Windows/HIP is untested so far; tell us if you cannot test them and we will.

[NektoDron]

I added paired GPU measurements with the grammar build option OFF and ON, and with an actual restrictive grammar in the ON build. Hardware: AMD EPYC 9654 (96 cores/192 threads), 624.9 GiB RAM reported by Linux, one RTX 5090 (32,607 MiB reported VRAM), NVIDIA driver 595.91.07; CUDA 13.1.115 compiler. The model was DeepSeek-V4.1-Flash-1017-heretic HXQ3K (custom XQ3_K expert quantization), with external FP8 Engram tables and DSpark auto, 64 CPU threads, native parallel=4 and a 524,288-token context. Requests were sequential, temperature=0, thinking and the Python loop guard disabled. Both binaries used the same Release/O3 build flags; only MAYA_DS4_GRAMMAR differed.

These are four fresh processes in OFF/ON/ON/OFF order, with three prompts per process and three additional grammar requests per ON process. Each measured request had 1,135–1,136 uncached prompt tokens and 130 output tokens. The grammar generated a complete JSON array of exactly 64 single decimal digits: all 6/6 outputs satisfied it.

Configuration/workload	Prefill tok/s, median (range)	Decode tok/s, median (range)
Grammar build OFF, no grammar	583.1 (578.6–592.5)	126.7 (107.1–130.0)
Grammar build ON, no grammar	592.2 (583.0–595.4)	111.7 (107.3–117.0)
Grammar build ON, restrictive grammar	580.0 (574.7–586.7)	90.1 (63.5–104.0)

The pooled ON/no-grammar decode figure is about 12% lower. However, the ON runs interleaved grammar requests, which changed the speculative/lookup history for later requests, so that number does not isolate the compile-option cost. The clean control is the first no-grammar request in each fresh process, before any grammar request: OFF decode was 108.4/107.1 tok/s and ON was 107.3/108.6 tok/s (medians 107.75/107.95); prefill was 587.7/592.5 versus 591.7/595.4 tok/s. All four produced byte-identical 130-token outputs and identical draft counts (131 offered, 116 accepted). This small control detected no slowdown, but does not establish zero overhead generally. Grammar does have a measurable decode cost in the constrained workload above.

DSpark verification ran on real model requests: the six grammar requests offered 700 drafts, with 578 accepted and 122 not accepted. The unconstrained totals were OFF 762/698/64 and ON 708/624/84 (offered/accepted/not accepted). Two of the three no-grammar prompt outputs matched across all four processes; the third matched within ON and within OFF but differed between the two mixed request histories. In the earlier automatic-selector experiment this third prompt also varied between repeated runs of the same build. These results do not certify universal no-grammar output identity in mixed histories, or establish the cause of a difference. A separate uninterrupted free-form repeat control showed variability without a sleep transition.

To keep prefill comparable I disabled the automatic prefill path selector only for these private benchmarks (DS4_PATH_PICK=0). An earlier automatic-selector run had one anomalously slow ON process (~159 tok/s prefill), while the next process with the same ON binary returned to ~582 tok/s; those observations were retained rather than folded into the fixed-path table.

Provenance: the GPU binaries are from the previously qualified integrated 73b4221 tree, including the F16 embedding loader needed by this pack and the sleep changes. They are not an exact standalone public-PR build. The grammar native code is identical to reviewed a469c09; the subsequent rebase has separate host regression coverage. I rebased the public branch onto current main 00655481; head 6fb4de7 passed 250 relevant Linux grammar/server/Responses/GLM API tests (2 skipped).

I cannot test Windows or HIP on this hardware, so please do run those on your setup. The committed Windows/Python 3.12 job runs the 16 grammar adapter/protocol tests; its actual hosted result still awaits workflow approval. Neither Windows nor HIP GPU execution is being claimed.

[mw00]

Thank you for the paired measurements and for separating the clean control from the interleaved runs. The first-request control shows no slowdown from the build option, and the constrained decode cost is clearly stated. One thing we would like to understand: the third no-grammar prompt differed between the OFF and ON mixed histories. Could you run OFF and ON with the same request order (grammar requests in the same places) to see if that output then matches? We will run Windows and HIP on our side.

[NektoDron]

I ran the request-order control. One clarification: OFF means the native build option MAYA_DS4_GRAMMAR=OFF; that binary rejects GBNF, so it cannot execute the same active grammar requests as ON. I therefore used two controls rather than count 400 responses as generation:

• Matched OFF/ON: four fresh processes in OFF/ON/ON/OFF order. Each received exactly the same six unconstrained requests in order P0, G0, P1, G1, P2, G2, with identical prompts/parameters and no grammar field in either build. The G labels identify the intervening prompts from the earlier grammar workload, not active constraints in this control.
• Mixed ON/ON: two further fresh ON processes received the same sequence with the grammar active at G0/G1/G2 in both. This repeats the literal mixed sequence in identical positions on a capable backend.

All 36 requests returned 200, read 1,135–1,136 uncached prompt tokens and produced 130 output tokens. Temperature was 0, thinking and loop guard off, DSpark on, private prefix directories separate per process, and DS4_PATH_PICK=0. The native pair and hardware are unchanged from the preceding report: integrated 73b4221, RTX 5090 / driver 595.91.07, EPYC 9654, 64 CPU threads, parallel 4 and context 524,288.

P0 and P1 were byte-identical across all six processes. P2 was not fully repeatable:

P2 run	Response SHA256 prefix
matched-off1	583987304259
matched-on1	583987304259
matched-on2	cd24668451bb
matched-off2	6891a3914a0c
mixed-on1	621b0badfbe2
mixed-on2	4aa67761b812

Thus P2 matched in the first OFF/ON pair, but differed in the second pair; importantly, it also differed between the two OFF runs and between the two ON runs despite identical request order. The two repeated mixed ON runs differed on P2 too; all six constrained G outputs still satisfied the grammar.

Keeping order identical therefore does not eliminate the observed variation. It can occur without any active GBNF request, including in the build with GBNF compiled out, so I cannot attribute it solely to mixed request order or the grammar option. This small control does not establish the numerical/root cause or certify universal no-grammar output identity. The raw results are retained; Windows/HIP execution remains on your side as offered.

[NektoDron]

I traced the unconstrained third-answer variation further. In this reproduction it starts as a numerical difference after adaptive expert placement, rather than a grammar mask or a random greedy sampler decision.

The fresh-process controls used the same warmup followed by six fixed unconstrained requests (P0/G0/P1/G1/P2/G2), temperature 0, thinking off and loop_guard: false, with separate cold prefix directories. Native grammar was compiled OFF in the qualified 73b4221 integration tree; this is not a standalone-current-main GPU claim. On the Q3_K trunk / MXFP4 DSpark fixture, CPU64, one RTX5090, --parallel 4, context524288:

• Ordinary adaptive expert cache: repeated processes can produce different P2 text.
• --no-expert-cache, with MTP still ON: all seven responses and all 776 recorded API logprob rows matched.
• Static placement with both MTP and suffix drafting OFF: the same equality held.

A subsequent pair captured native target logits, expert residency and layer states. The pair had identical initial placement and an identical warmup; its first 17 P0 logit rows were bit-identical. The first residency difference was expert210 on layer4, GPU versus CPU, with the same selected expert IDs and verification-window shape. The first captured hidden-state difference immediately afterwards was only 2.98e-8 (relative L2 about 2.95e-8). It then amplified: the first differing target-logit row differed by up to 0.515 while the emitted history still matched. Text diverged later. The layer capture is after attention, so the first hidden-state bracket includes MoE4 and the next attention layer rather than a direct expert-output measurement.

All 851 byte-for-byte expert-cache checks in that matched diagnostic pair passed; there were no copy failures, mismatches or CUDA errors. CPU/GPU expert execution and reduction grouping differ, so timing-dependent adaptive residency can alter the last bits. Later distribution changes can be substantial; this is not merely an almost-tied top-2 choice. Freezing placement removed the variation for this workload, but is not a general cross-hardware determinism guarantee.

One diagnostic pair was excluded from the matched-start comparison: its first launch calibrated PCIe after allocating experts (970 in VRAM), while the next read the saved calibration before allocation (1018). Repeating with the same cached calibration removed that startup confound. Trace synchronization affects timing, so none of these diagnostic timings are speed measurements.

Separately, I found and fixed a failure-path bug where a failed GGUF expert move was still published. See #935. It was not encountered in the successful-run nondeterminism tests and should not be conflated with their numerical origin.

[NektoDron]

Rebased onto main 6651b5b (v1.1.63), with head be709c1df779eebf39c6f5d86a79c34e4111869f.

The current mixed-GPU/pool changes have been retained. A linked pool or split layout samples in its other engine, so the grammar capability is zero there and an explicit grammar is refused before request state changes. Grammar stays available for the local sampling head, including the mixed-GPU helper layout and native parallel decoding.

The rebase also keeps grammar exhaustion on the existing DONE grammar path when the native -2 sentinel is returned. CPU regressions check frontend rejection, native capability reporting, request ordering and the pool fallback distinction.

Public-head Linux checks passed with 978 server tests (40 skipped), 941 installer/tools tests (40 skipped), all 18 focused grammar tests and the HIP mapping check.

An additional optional Transformers float64 oracle check failed with the local Torch/Transformers stack because grouped matrix multiplication rejects Double input. The same failure was reproduced on unchanged main 6651b5b. The tools suites passed using the dependency set from the existing CI workflow, where this optional check is skipped.

The standalone public head passed CUDA Release builds with the default grammar option OFF and an explicit ON build. The default target graph excludes the grammar library and checker. Metadata/vocabulary grammar-state and path checks passed. Build tools were CUDA 13.1, GCC 14 and SM120.

Fresh integrated RTX 5090 checks at 728cd8c5 passed 24 grammar cases and 20 sleep cases, covering four native request slots, grammar changes with prefix reuse, cancellation and follow-up, busy exclusion, explicit wake and concurrent automatic wake. The fixture combines the current #855 and #856 source with the exact #854 F16 loader required by the custom pack. The standalone public #856 CUDA build was checked separately.

Actual speculative execution was observed in the stochastic grammar probe with draft_n=1 and draft_n_accepted=1. The marker restored with 3615 cached tokens. The observed checkpoint took 14.364 s and explicit wake took 9.225 s. These are feature-smoke observations on this fixture.

Each OFF/ON build accepted 20 legacy-field requests across both APIs and streaming modes. OFF rejected explicit native GBNF with 400, and ON accepted the nullable-sibling grammar request. Comparing the legacy outputs, visible text matched in 19/20 pairs, reasoning text in 12/20, and both together in 11/20. Legacy JSON parsing succeeded for 13/20 OFF responses and 14/20 ON responses. Six OpenAI pairs returned the same invalid {"ok":<|end|>true} text in both builds. One additional OFF OpenAI streaming json_schema response had that marker, while its ON counterpart was valid JSON with a newline. All ten Anthropic final responses were valid JSON, and eight of those pairs had different reasoning text. The route checks establish HTTP/protocol acceptance.

MAYA_DS4_GRAMMAR remains OFF by default. Existing JSON aliases retain their handling. The integrated GPU fixture includes #854 for its embedding storage and #855 for sleep checks. Earlier performance and request-order observations remain in the description with their original source scope. Hosted Windows execution and HIP hardware qualification remain pending.