feat(speculative): n-gram drafter stacked on MTP for Qwen3.6-35B-A3B

- New per-request n-gram speculative drafter with <think> and <tool_call>
  state machines, adaptive K based on n-gram match confidence, hybrid
  verify (append MTP draft after n-gram tail), per-request self-tuning,
  and global auto-disable when MTP is strong and n-gram is weak.
- Auto-enabled by the qwen3.6-35b preset and the new qwen3.6-35b-8bit
  preset. +18% throughput on agentic reasoning + tool-use workloads vs.
  MTP-only.
- New qwen3.6-35b-8bit alias routing to
  samuelfaj/Qwen3.6-35B-A3B-8bit-MTPLX-Optimized-Speed with full preset
  parity (MTP, n-gram, port 8010, tool/reasoning parsers, temps).
- Structured CoT grammar plumbing (structured_cot.gbnf, lcb_plan.gbnf).
- Scheduler, server, TUI, and metrics middleware updates to surface and
  control n-gram drafting per request.
- Test coverage for drafter, structured CoT, and CLI preset parity.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Author: samuelfaj

README.md

@@ -48,6 +48,19 @@ create the snake game using react and typescript
 
 You can check for more benchmarks (for non-optmized models) in [Rapid-MLX](https://github.com/raullenchai/Rapid-MLX).
 
+## N-gram Speculation (Qwen3.6-35B-A3B)
+
+On the 35B-A3B preset, n-gram (prompt-lookup) drafting is layered on top of MTP for **+18% throughput** on mixed reasoning + tool-use workloads (vs. MTP-only). Auto-enabled for `qwen3.6-35b` and `qwen3.6-35b-8bit`.
+
+Highlights:
+
+- **`<think>`-aware** and **`<tool_call>`-aware** state machines: drafts everywhere by default but skips inside `<tool_call>...</tool_call>` regions where structure repeats but content varies.
+- **Adaptive K** based on n-gram match confidence (prior occurrence count): wide drafts for strong matches, narrow drafts for weak ones.
+- **Hybrid verify**: append one MTP-head draft after the n-gram tail to capture extra ground when n-gram drafts all accept.
+- **Self-tuning**: per-request running acceptance suppresses drafting on bad fits; global auto-disable when MTP is already strong (≥0.85) and n-gram is weak (≤0.50). Guarantees no regression vs. the MTP-only baseline.
+
+Tunable via `--enable-ngram` / `--disable-ngram` and `--ngram-*` flags on `lightning-mlx serve` and `bench`.
+
 ## Install
 
 ```bash
@@ -81,8 +94,11 @@ Best optimized models:
 ```bash
 lightning-mlx serve qwen3.6-27b
 lightning-mlx serve qwen3.6-35b
+lightning-mlx serve qwen3.6-35b-8bit
 ```
 
+`qwen3.6-35b-8bit` mirrors the `qwen3.6-35b` preset (MTP, n-gram, port 8010, tool/reasoning parsers) but routes to the 8-bit MTPLX-optimized weights for higher quality on memory-rich Macs.
+
 Local model path works too:
 
 ```bash
@@ -147,6 +163,7 @@ curl http://localhost:8010/v1/chat/completions \
 
 - **2.75x faster short agentic turns** in the benchmark fixture.
 - **1.96x higher all-turn throughput** versus the MLX baseline.
+- **+18% throughput on Qwen3.6-35B-A3B** with n-gram + MTP stacked speculation.
 - **Successful artifact generation** where baseline timed out.
 - **OpenAI-compatible API** for local tools, agents, editors, and CLIs.
 - **Apple Silicon first**: built around MLX and local Mac inference.

tests/test_mtplx_cli_preset.py

@@ -1,8 +1,10 @@
 import argparse
 
 from vllm_mlx.cli import (
+    _QWEN36_35B_8BIT_MTPLX_MODEL,
     _QWEN36_35B_MTPLX_MODEL,
     _QWEN36_MTPLX_MODEL,
+    _apply_qwen36_35b_defaults,
     _apply_qwen36_mtplx_preset,
 )
 from vllm_mlx.scheduler import SchedulerConfig
@@ -32,6 +34,21 @@ def _serve_args(**overrides):
         "no_thinking": False,
         "log_level": "INFO",
         "enable_tool_logits_bias": False,
+        # N-gram defaults (preset overrides these for 35B-A3B).
+        "enable_ngram": False,
+        "ngram_num_draft_tokens": 4,
+        "ngram_size": 3,
+        "ngram_min_matches": 2,
+        "ngram_only_in_think": True,
+        "ngram_acceptance_mode": "greedy",
+        "ngram_min_occurrences": 1,
+        "ngram_adaptive_k": True,
+        "ngram_auto_disable_mtp_threshold": 0.0,
+        "ngram_auto_disable_min_ngram": 0.50,
+        "ngram_hybrid_verify": False,
+        "ngram_skip_tool_calls": True,
+        "ngram_self_tune": True,
+        "ngram_self_tune_disable_threshold": 0.30,
     }
     values.update(overrides)
     return argparse.Namespace(**values)
@@ -144,3 +161,115 @@ def test_qwen36_mtplx_preset_keeps_explicit_prefill_step_size():
 
 def test_scheduler_default_prefill_step_size_is_sustained():
     assert SchedulerConfig().prefill_step_size == 8192
+
+
+def test_qwen36_35b_serve_preset_enables_ngram_with_tuned_defaults():
+    args = _serve_args(model=_QWEN36_35B_MTPLX_MODEL)
+
+    _apply_qwen36_mtplx_preset(args, ["serve", _QWEN36_35B_MTPLX_MODEL])
+
+    # N-gram is auto-enabled for 35B-A3B with the validated agentic
+    # configuration.
+    assert args.enable_ngram is True
+    assert args.ngram_num_draft_tokens == 6
+    assert args.ngram_min_occurrences == 2
+    assert args.ngram_acceptance_mode == "greedy"
+    assert args.ngram_hybrid_verify is True
+    assert args.ngram_only_in_think is False  # everywhere
+    assert args.ngram_skip_tool_calls is True
+    assert args.ngram_self_tune is True
+    assert args.ngram_self_tune_disable_threshold == 0.30
+    assert args.ngram_auto_disable_mtp_threshold == 0.85
+    assert args.ngram_auto_disable_min_ngram == 0.50
+
+
+def test_qwen36_35b_serve_preset_disable_ngram_flag_overrides():
+    args = _serve_args(model=_QWEN36_35B_MTPLX_MODEL)
+
+    _apply_qwen36_mtplx_preset(
+        args,
+        ["serve", _QWEN36_35B_MTPLX_MODEL, "--disable-ngram"],
+    )
+
+    assert args.enable_ngram is False
+
+
+def test_qwen36_35b_serve_preset_keeps_explicit_ngram_overrides():
+    args = _serve_args(
+        model=_QWEN36_35B_MTPLX_MODEL,
+        ngram_num_draft_tokens=8,
+        ngram_min_occurrences=4,
+        ngram_hybrid_verify=False,
+    )
+
+    _apply_qwen36_mtplx_preset(
+        args,
+        [
+            "serve",
+            _QWEN36_35B_MTPLX_MODEL,
+            "--ngram-num-draft-tokens",
+            "8",
+            "--ngram-min-occurrences",
+            "4",
+        ],
+    )
+
+    assert args.ngram_num_draft_tokens == 8
+    assert args.ngram_min_occurrences == 4
+    # User did NOT pass --ngram-hybrid-verify, so the preset still flips
+    # it on (the existing hybrid_verify=False in args is the parser's
+    # default, not an explicit override).
+    assert args.ngram_hybrid_verify is True
+
+
+def test_qwen36_35b_serve_preset_no_hybrid_verify_overrides():
+    args = _serve_args(
+        model=_QWEN36_35B_MTPLX_MODEL,
+        ngram_hybrid_verify=False,
+    )
+
+    _apply_qwen36_mtplx_preset(
+        args,
+        [
+            "serve",
+            _QWEN36_35B_MTPLX_MODEL,
+            "--no-ngram-hybrid-verify",
+        ],
+    )
+
+    assert args.ngram_hybrid_verify is False
+
+
+def test_qwen36_35b_8bit_alias_matches_4bit_preset():
+    """8bit alias must apply identical defaults — only model differs."""
+    a = _serve_args(
+        model=_QWEN36_35B_MTPLX_MODEL, _original_alias="qwen3.6-35b"
+    )
+    b = _serve_args(
+        model=_QWEN36_35B_8BIT_MTPLX_MODEL,
+        _original_alias="qwen3.6-35b-8bit",
+    )
+
+    _apply_qwen36_mtplx_preset(a, ["serve", "qwen3.6-35b"])
+    _apply_qwen36_35b_defaults(a, ["serve", "qwen3.6-35b"])
+    _apply_qwen36_mtplx_preset(b, ["serve", "qwen3.6-35b-8bit"])
+    _apply_qwen36_35b_defaults(b, ["serve", "qwen3.6-35b-8bit"])
+
+    da, db = vars(a), vars(b)
+    diffs = {k: (da[k], db[k]) for k in da if da[k] != db[k]}
+    assert diffs == {
+        "model": (_QWEN36_35B_MTPLX_MODEL, _QWEN36_35B_8BIT_MTPLX_MODEL),
+        "_original_alias": ("qwen3.6-35b", "qwen3.6-35b-8bit"),
+    }
+
+
+def test_qwen36_27b_serve_preset_does_not_enable_ngram():
+    """27B model should not get the 35B-only ngram preset."""
+    args = _serve_args(model=_QWEN36_MTPLX_MODEL)
+
+    _apply_qwen36_mtplx_preset(args, ["serve", _QWEN36_MTPLX_MODEL])
+
+    assert args.enable_ngram is False
+    assert args.ngram_num_draft_tokens == 4  # parser default unchanged
+    assert args.ngram_min_occurrences == 1
+    assert args.ngram_hybrid_verify is False