Skip to content

qwen3.6-27b-text-nvfp4-mtp.md

Latest commit

 

History

History
408 lines (360 loc) · 24.7 KB

qwen3.6-27b-text-nvfp4-mtp.md

File metadata and controls

408 lines (360 loc) · 24.7 KB
Raw

Default primary: sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP

The fleet's default primary since 2026-05-31 — a text-only MTP build: the 27B re-exported so vLLM speculative decoding (Multi-Token Prediction) actually works. The archived baseline (mmangkad/Qwen3.6-27B-NVFP4, qwen3.6-27b-nvfp4.md) is slow single-stream on the GB10 (~8 tok/s decode); MTP drafts several tokens per forward pass and roughly doubles that. Promoted from candidate after the tool-calling gate passed (see "Tool calling — verified" below). Filed as issue #26.

Status: the fleet default primary (promoted 2026-05-31 on this GB10). Loaded and served on spark-f8a9 (vLLM 0.19.0+nv26.04) at 18.7–19.1 tok/s decode (~2.4× the archived baseline 27B's ~8 tok/s) with 72–79 % MTP draft acceptance, and the tool-calling gate passed under the production compose (valid qwen3_coder tool call + full round-trip + reasoning trace, MTP active) — see the benchmark and "Tool calling — verified" sections below. One packaging fix is needed (a tokenizer override — caveat 1), baked into the compose template. For the catalog-vs-warm distinction — what you can load vs. what's loaded now — see gateway-fleet.md.

Now serving on vLLM nightly, 64K fleet default (since t4, 2026-07-01 — docs/vllm-nightly-migration.md §4/§8). The fleet flipped this gear (plus embed/rerank) from the NGC 26.04-py3 image (vLLM 0.19.0+nv26.04, the engine every benchmark below was measured on) to the same pinned vLLM-nightly digest (0.23.1rc1.dev672) the Gemma multimodal gear already ran — one engine, fleet-wide. Nightly auto-maps the deprecated qwen3_5_mtp method string to mtp with MTP staying active (69.9 % draft acceptance on the t2 spike, near the 72–79 % below; a same-engine head-to-head later measured 60.6 %, §6) — no caller/config change needed. The fleet-default context is also now 64K, not the 256K described below: PRIMARY_MAX_MODEL_LEN=65536 / PRIMARY_GPU_MEM_UTIL=0.30, trimmed so the always-on Gemma 4 12B multimodal gear can co-reside at its full 128K native context (docs/vllm-nightly-migration.md §8; see gateway-fleet.md for the duo budget). The benchmarks below (0.19.0 engine, 32K/128K/256K context) remain the historical promotion record, not re-measured at 64K/nightly here — see docs/vllm-nightly-migration.md §4/§6 for the nightly-engine numbers.

Source: https://huggingface.co/sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP.

Why a different checkpoint at all (the 35B lesson)

We tried to enable MTP the obvious way first and learned the constraint is the checkpoint, not the engine — the same wall the 35B MoE hit:

  • A baseline NVFP4 export drops the MTP draft head. Both the original unsloth/Qwen3.6-27B-NVFP4 and mmangkad/Qwen3.6-27B-NVFP4 are plain NVFP4 quantizations; per the model cards the MTP layer is discarded during export, so vLLM speculative decoding gets ~0 % draft acceptance (and the 35B mmangkad/ copy fails to load the draft outright — qwen3_5_mtp.py weight-shape mismatch).
  • The fix the 35B doc points at — a newer vLLM with the right loader — is not installable on this aarch64 GB10 yet (cutlass-dsl conflicts ≥ 0.22; the image stays NGC 26.04-py3 / vLLM 0.19.0).

So the working path is the other half of the lesson: serve a checkpoint that ships the MTP draft weights. sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP is the 27B with its MTP head grafted back in bf16 (~850 MB, 15 tensors) specifically for vLLM — no newer engine required.

What it is

  • An NVFP4 (NVIDIA ModelOpt) re-export of Qwen/Qwen3.6-27B with the MTP draft head restored in bf16 for vLLM speculative decoding (the baseline export dropped it). 27.78B params, 64 layers + 1 MTP layer, 256K native context.
  • Text-only: the ViT vision tower is physically deleted from this build, so it serves with --language-model-only and AutoModelForCausalLM (no image path). The fleet runs the 27B text-only anyway, so nothing is lost for our use.
  • Quantization is modelopt (not modelopt_fp4) per the card — NVFP4 on the native SM120 fast path. lm_head, the Mamba conv1d SSM convolutions, and all mtp.* modules are kept in bf16. The catalog sets VLLM_QUANTIZATION=modelopt; verify on first load (if vLLM rejects it, try modelopt_fp4).

Is it supported here? — yes, confirmed (2026-05-31)

The MTP method this checkpoint declares is qwen3_5_mtp, and vLLM 0.19.0+nv26.04 accepts it — this was the open risk, now resolved. On load the engine logs:

WARNING [speculative.py:368] method `qwen3_5_mtp` is deprecated and replaced with mtp.
INFO    [model.py:549] Resolved architecture: Qwen3_5ForConditionalGeneration
INFO    [model.py:549] Resolved architecture: Qwen3_5MTP
INFO    [core.py:105] ... speculative_config=SpeculativeConfig(method='mtp', num_spec_tokens=3)

Both the target (Qwen3_5ForConditionalGeneration) and the draft head (Qwen3_5MTP) resolve on the stock image — no newer engine needed. Live draft acceptance came out at 72 % (see the benchmark table).

How to run (compose edits lobes switch prints)

lobes switch writes the VLLM_* keys to .env, but the MTP draft and the text-only flags can't be defaulted in the shared template (compose can't omit an empty flag, and they break the dense/hybrid models). So lobes switch resolves the env and then prints the exact compose command: list items to add by hand — the same mechanism the MoE --moe-backend uses. The compose command: is a YAML list (one argv token per item), so each flag is a separate item and the --speculative-config JSON uses the = form, single-quoted (its value contains { and : characters):

# add to the vLLM service `command:` list in docker-compose.yml
      - '--speculative-config={"method": "qwen3_5_mtp", "num_speculative_tokens": 3}'
      - --trust-remote-code
      - --language-model-only
      - --tokenizer=mmangkad/Qwen3.6-27B-NVFP4

lobes switch sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP --apply also auto-selects VLLM_TOOL_CALL_PARSER=qwen3_coder + VLLM_QUANTIZATION=modelopt; separately set VLLM_MAX_NUM_SEQS=2 in .env (see the OOM caveat below).

Reference serve recipe (from the card) once those lines are in the compose file:

# Production (256K context, KV FP8, MTP n=3)
vllm serve sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP \
  --trust-remote-code --language-model-only \
  --quantization modelopt --kv-cache-dtype fp8 \
  --max-model-len 262144 --max-num-seqs 2 \
  --gpu-memory-utilization 0.9 --reasoning-parser qwen3 \
  --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'

# Smaller context (16K) for the fastest single-request decode — same flags,
#   --max-model-len 16384  (and drop --kv-cache-dtype fp8 if it OOMs on warmup)

On the shared GB10 keep --gpu-memory-utilization at the machine profile's 0.6 (not 0.9 — the audio NIMs + reachy hold the rest of the 121.7 GiB). The served default is now the full 256K (--max-model-len 262144), load-tested 2026-06-03 (see the 256K benchmark below): same ~70 GiB footprint as 32K/128K at util 0.6 — --gpu-memory-utilization fixes the KV-pool reservation, so context length doesn't change resident memory; only the addressable window grows. vLLM reports 5.3× max concurrency at a full 256K request — well above the seqs=2 decode cap, so there is no practical concurrency loss. Lower the context only if heavier co-resident agents need the headroom.

Caveats — validated on the live load (2026-05-31)

  1. Tokenizer override required (load-blocking). The checkpoint's tokenizer_config.json declares "tokenizer_class": "TokenizersBackend", a newer-transformers class absent from the nv26.04 image — vLLM crash-loops with ValueError: Tokenizer class TokenizersBackend does not exist or is not currently imported. Fix: point vLLM at the sibling base tokenizer (same Qwen3.6-27B vocab, already cached in production) — add --tokenizer=mmangkad/Qwen3.6-27B-NVFP4 to the compose command. With that it loads cleanly. (lobes switch prints this line.)

  2. --max-num-seqs 2 is load-bearing. The card warns that --max-num-seqs 4 with KV-FP8, n=3, and 256K context silently OOMs during CUDA-graph capture. No --purpose profile yields 2 (balanced/prompt-heavy=4, decode-heavy=8), so set VLLM_MAX_NUM_SEQS=2 in .env by hand (lobes switch to this primary forces it). Tested at 32768 context / util 0.6 on the shared box (~71.5 GiB resident); the full 256K (262144) served default was load-tested too (2026-06-03) — boots clean at seqs=2 (the seqs=2 cap is exactly what keeps the capture OOM off at full context), same ~70 GiB footprint. The OOM warning above is specifically about seqs=4.

  3. Quantization. --quantization modelopt works — vLLM resolves it to modelopt_fp4 (NVFP4) on load. The catalog sets modelopt.

  4. --trust-remote-code + --language-model-only are required (custom modeling code; vision tower removed) — the shared template omits both, hence the compose edits.

  5. MTP + reasoning + structured output. This build serves with --reasoning-parser qwen3 (the reasoning trace field populated correctly in testing); MTP has a known vLLM edge case where speculation can break </think> detection under structured output (vLLM #34650). Watch JSON-schema responses.

  6. preserve_thinking — thinking-history continuity (issue #93). The compose command: also adds --default-chat-template-kwargs '{"preserve_thinking": true}' next to --reasoning-parser=qwen3. preserve_thinking is a real variable in the served mmangkad/Qwen3.6-27B-NVFP4 chat template: by default the template keeps only the <think> block(s) after the last user turn when it renders a multi-turn history; with the flag set, it keeps every historical <think> block, so a later turn's prompt includes the model's earlier reasoning, not just its final answers. It is a server-side default, not a hard override — a request's own chat_template_kwargs (e.g. {"preserve_thinking": false}) still wins, so lobes route's enable_thinking=false routing path is unaffected. This is default-on for the cortex/main generate lane only; the embed/rerank/senses lanes don't take this flag and are untouched.

    • Read-only diagnostic. A two-turn prompt-token-count diagnostic proves the round-trip is live without mutating anything: send the same two-turn conversation once with reasoning history included and once with content-only history, and compare usage.prompt_tokens on the second turn — with preserve_thinking on, the prompt token count is measurably higher (the server re-sent the earlier <think> block(s) back into context). This shows the input side round-trips correctly; it does not by itself measure whether the extra reasoning context improves the model's answers — that continuity benefit is expected and opt-in, not a guarantee the diagnostic asserts.

  7. Tool calling — verified (2026-05-31), gate closed. The first run's minimal recipe omitted --enable-auto-tool-choice; the promotion run served through the compose template (which enables it + --tool-call-parser=qwen3_coder) and the tool path passed — see "Tool calling — verified" below.

  8. Strict tool calling with thinking — think-aware structural tags (colleague#320). Unconstrained thinking-mode generation can drift off the qwen3_coder XML tool-call template — a captured request reproduced a hallucinated <tool_call>\n\n[tool_id]{json} emission, deterministic at temperature 0 with that request, and not a long-prompt effect (a ~4K-char system prompt sufficed) — and vLLM's parser then "salvages" a mangled call (name='read_file"', empty arguments). OpenAI's strict: true on a tool's function arms xgrammar structural-tag constrained decoding (VLLM_ENFORCE_STRICT_TOOL_CALLING defaults true, but with tool_choice:"auto" it only arms when a tool itself declares strict), which makes a malformed call impossible by construction — but the served build (0.23.1rc1.dev672) hardcodes reasoning=False at the structural-tag call site (vllm/parser/abstract_parser.py: _apply_structural_tag), so with thinking enabled the resulting grammar cannot accept the </think> special token and a strict request 500s ("grammar rejected tokens"). With enable_thinking:false, strict already returns a clean call (MTP spec-decode stays active throughout). The reasoning-aware grammar exists in the image — only the call site never passes reasoning=True.

    The fix is the qwen3_coder_thinking tool-parser plugin (lobes/vllm_plugins/qwen3_thinking_tool_parser.py, registered under the name qwen3_coder_thinking via vLLM's own --tool-parser-plugin file-path surface — vLLM execs the file directly, it is never import lobes): it overrides get_structural_tag to derive reasoning from the request's own effective enable_thinking (absent → on, the server default — same rule preserve_thinking above uses) instead of the caller's hardcoded False, so lobes route's enable_thinking=false terse path keeps getting a thinking-free grammar, unaffected. It is the compose default (PRIMARY_TOOL_CALL_PARSER=qwen3_coder_thinking, falling back to plain upstream qwen3_coder if an operator overrides it), loaded on the cortex/main generate lane only (mirrors the preserve_thinking #93 scoping — the multimodal/embed/rerank lanes never mount or load it), and asserts loudly at import time if the served image's tool-parser surface ever drifts from the pinned 0.23.1rc1.dev672 shape, refusing to boot rather than silently serving an unconstrained grammar. No vLLM image rebuild (the pin is unchanged) and no served-model change — this constrains around the template drift, it does not fix the drift itself (out of scope here; filing the reasoning=False hardcode upstream is a tracked follow-up). A companion, independent gateway knob (GATEWAY_FORCE_STRICT_TOOLS) that injects strict: true for existing callers who don't set it themselves is documented in docs/openai-api.md and docs/gateway-fleet.md. Verified at spec time by replaying the captured request across the four thinking×strict combinations; fleet-level acceptance runs in the deployment, as a separate follow-on task.

Live-test (run on spark-f8a9, 2026-05-31) — how to reproduce

  1. Premise (unsloth baseline). The ticket (#26) names a baseline serve command — vllm serve unsloth/Qwen3.6-27B-NVFP4 --trust-remote-code --dtype bfloat16 --max-model-len 4096. We deliberately deviate: that baseline NVFP4 export drops the MTP head → ~0 % draft acceptance (established by the unsloth + sakamakismile model cards), so running it cannot exercise MTP — which is the whole point. It was not re-run live to avoid a redundant ~28 GB download. The decode comparison below is against the already-load-tested mmangkad/Qwen3.6-27B-NVFP4 (same arch + NVFP4, the archived primary), not the unsloth artifact. The grafted re-export is what makes MTP work.
  2. The real test (sakamakismile). Stop the primary first (one ~30B model fits on the GB10 at a time), serve with the compose edits above + VLLM_MAX_NUM_SEQS=2, docker compose up -d → /health, then lobes assess (correctness) + lobes benchmark (decode tok/s), plus GET /metrics (vllm:spec_decode_num_accepted_tokens_total / …_draft_tokens_total) for the draft-acceptance rate. Result: healthy, 19.1 tok/s, 72 % acceptance (below).
  3. Risk gate — passed. vLLM 0.19.0+nv26.04 accepts qwen3_5_mtp, so no newer engine is needed.

Benchmark — 2026-05-31, DGX Spark (GB10), shared box

Image nvcr.io/nvidia/vllm:26.04-py3 (vLLM 0.19.0+nv26.04), served on :8001 at --max-model-len 32768 --gpu-memory-utilization 0.6 --max-num-seqs 2 --kv-cache-dtype fp8 with --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3}' and --tokenizer=mmangkad/Qwen3.6-27B-NVFP4.

Property Value
Health / max_model_len /health 200; 32768
Engine accepts qwen3_5_mtp ✅ resolves Qwen3_5MTP draft head; method → mtp
Correctness 17 × 23 = 391 ✅ (finish=stop, 384 tok); reasoning trace reasoning (4,817 chars)
Decode throughput 19.1 / 19.1 tok/s (batch=1, greedy, 512 tok forced)
Prefill 845 prompt tokens + 16 gen in 1.23 s
MTP draft acceptance 72.2 % (732 / 1,014 draft tokens; ~2.17 of 3 accepted per step)
GPU memory reserved ~71.5 GB (71,525 MiB) at util 0.6
Tool calling not exercised (minimal recipe omitted --enable-auto-tool-choice)

Comparison — baseline 27B primary (same box)

MTP (this checkpoint) baseline mmangkad/Qwen3.6-27B-NVFP4
Decode (batch=1, 512 tok) 19.1 tok/s 7.8–8.0 tok/s
Prefill (~845 tok) 1.23 s 2.33 s
Speedup ~2.4× decode —

So MTP gives a ~2.4× single-stream decode speedup on the GB10 — above the card's 1.74× (which was concurrency 2 on a Blackwell box; single-stream with 72 % acceptance is MTP's best case). One assess probe (the 145-min word problem) hit finish=length at the 2,048-token cap mid-reasoning — a reasoning-verbosity artifact, not a wrong answer.

Benchmark — 2026-06-03, 128K context (the new served default)

Re-tested on the same shared GB10 at the 128K default (--max-model-len 131072), otherwise the same image and flags as the 2026-05-31 run (util 0.6, --max-num-seqs 2, KV-FP8, MTP n=3, --tokenizer=mmangkad/Qwen3.6-27B-NVFP4). Reproduce with lobes switch sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP --max-model-len 131072 --apply, then lobes assess + lobes benchmark.

Property Value
Health / max_model_len /health 200; 131072
Boots clean (no capture OOM) ✅ CUDA-graph pool 0.49 GiB; loaded in 6m46s
KV cache 48.39 GiB available; 372,800 tokens; 9.61× max concurrency at 131,072 tokens/request
Correctness 17 × 23 = 391 ✅ (391 tok); 14:45→17:10 = 145 min ✅ (692 tok, completes — no finish=length this time); reasoning trace reasoning (1,639 chars)
Decode throughput 18.3 tok/s (batch=1, greedy, 1000 tok forced)
Prefill 845 prompt tokens + 16 gen in 1.22 s
MTP draft acceptance 73.3 % (2,176 / 2,967 draft tokens; ~2.20 of 3 accepted per step)
GPU memory (EngineCore) 71,963 MiB (~70 GiB) at util 0.6 — same as 32K
Tool calling ✅ post-switch probe passed (tool_choice:"auto", finish=tool_calls)

The 2026-05-31 32K table above remains the original promotion measurement; 128K matches it on throughput, draft acceptance, and footprint. Because util fixes the KV-pool reservation, the resident footprint is unchanged from 32K — only the addressable context grows (and the pool still holds 9.6× a full 128K request, so there is room to push higher with headroom). That headroom was then spent: 256K is the served default since 2026-06-03 (next section).

Benchmark — 2026-06-03, 256K context (the new served default)

Re-tested on the same shared GB10 at the full 256K native context (--max-model-len 262144), otherwise the same image and flags as the 128K run (util 0.6, --max-num-seqs 2, KV-FP8, MTP n=3, --tokenizer=mmangkad/Qwen3.6-27B-NVFP4). Reproduce with lobes switch sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP --max-model-len 262144 --apply, then lobes assess --tools + lobes benchmark.

Property Value
Health / max_model_len /health 200; 262144
Boots clean (no capture OOM) ✅ CUDA-graph capture (PIECEWISE) 0.71 GiB in 2 s
KV cache 49.14 GiB available; 377,600 tokens; 5.29× max concurrency at 262,144 tokens/request
Correctness 17 × 23 = 391 ✅ (329 tok); 14:45→17:10 = 145 min ✅ (701 tok, completes); reasoning trace reasoning (1,589 chars)
Decode throughput 17.8 tok/s (batch=1, greedy, 1000 tok forced)
Prefill 845 prompt tokens + 16 gen in 1.25 s
MTP draft acceptance 74.0 % (2,282 / 3,084 draft tokens; ~2.22 of 3 accepted per step)
GPU memory (EngineCore) 71,601 MiB (~70 GiB) at util 0.6 — same as 32K/128K
Tool calling ✅ post-switch probe passed (tool_choice:"auto", finish=tool_calls)

256K matches the 32K and 128K runs on footprint, throughput (the gentle context-decline 19.1→18.3→17.8 tok/s continues), and draft acceptance (~74 %), and adds the full native window at no extra resident memory — --gpu-memory-utilization fixes the KV-pool reservation, so only the addressable context grows. The KV pool gives 5.29× concurrency at a full 256K request, well above the --max-num-seqs 2 decode cap, so there is no practical concurrency cost vs the 128K default. The seqs=2 cap is what holds off the capture OOM the seqs=4 caveat warns about.

Boot note: one engine-core attempt hit a transient InductorError: CUDA driver error: operation not permitted during torch-inductor autotuning (a CUDA forward-compatibility-mode artifact on this box — the driver is in compat mode), after weights loaded. It is not an OOM and not context-related; the engine retried and booted clean (adding ~3 min to the boot). Watch for it on a cold boot, but it self-recovers.

Tool calling — verified (2026-05-31)

The promotion gate was tool calling: the mesh agent rides on this model over the acp backend and uses tool calls, so MTP speculative decoding must not break the tool path. Served through the production compose (so the flags the original minimal recipe omitted were all present: --enable-auto-tool-choice, --tool-call-parser=qwen3_coder, plus --async-scheduling / --enable-chunked-prefill / --enable-prefix-caching), at --max-model-len 32768 --gpu-memory-utilization 0.6 --max-num-seqs 2:

Gate Result
Tool call emission (tool_choice:"auto") ✅ valid qwen3_coder call get_weather({"city":"Tokyo"}), parseable JSON
Full tool round-trip (result → final answer) ✅ finish=stop, correct natural-language answer
Reasoning + tool calling coexist (vLLM #34650) ✅ reasoning trace present, no </think> breakage
MTP spec-decode active with tools on ✅ 78.6 % draft acceptance (276/351) — not silently disabled
Correctness (lobes assess) ✅ both probes finish=stop (the 145-min word problem that hit length in the minimal run now completes)
Decode throughput (lobes benchmark, production flags) ✅ 18.7 tok/s — the ~2.4× win survives the full flag set
GPU footprint ~71.2 GiB (72,915 MiB) at util 0.6 — fits the shared box

The one packaging caveat (the --tokenizer override, caveat 1) is handled by baking it into the compose template, so a fresh deploy works out of the box. Reproduce with lobes assess + lobes benchmark and a tool_choice:"auto" request (or lobes switch sakamakismile/Qwen3.6-27B-Text-NVFP4-MTP --apply, which runs the tool-call probe automatically).

Recommendation

Promoted to fleet default primary (2026-05-31) — both gates the earlier draft called out are closed: tool calling is verified (above) and the tokenizer override is handled in the template. The win: ~2.4× single-stream decode (8 → ~19 tok/s) at 72–79 % MTP acceptance, same ~71 GB footprint, on the stock nv26.04 image; text-only is no loss (the fleet runs the 27B text-only anyway). The archived baseline mmangkad/Qwen3.6-27B-NVFP4 stays a candidate (the tokenizer source here and the only vision-capable 27B). Ideally fix the checkpoint's tokenizer_config.json upstream so no --tokenizer override is needed. The same MTP-grafted-checkpoint strategy is the path to MTP on the 35B (see the 35B follow-up).

This gear has since moved off the stock nv26.04 image onto the fleet-wide vLLM nightly digest (t4, docs/vllm-nightly-migration.md §4) and its fleet-default context trimmed to 64K to share the box with the always-on Gemma multimodal gear (§8) — MTP survives both changes unchanged (the deprecated method string auto-maps, no config edit needed); see the status note at the top of this doc.