Add tutorial: Qwen3.6-27B with MTP (Multi-Token Prediction) on Jetson Thor / AGX Orin for local agentic coding

[tokk-nv]

Community signal

Multiple high-engagement posts this week show massive interest in Qwen3.6-27B with MTP (Multi-Token Prediction) for local agentic coding:

• 2.5x faster inference with Qwen 3.6 27B using MTP — 596 upvotes, 186 comments
• Qwen3.6-27B MTP grafted on Unsloth UD XL: 2.5x throughput — 89 upvotes
• Qwen 3.6 27B MTP on V100 32GB: 54 t/s — 69 upvotes
• Quality comparison of Qwen 3.6 27B quantizations — 383 upvotes

The narrative is clear: Qwen3.6-27B + MTP + a 48GB budget = first viable local replacement for Claude Code / Codex at 262k context.

Why this matters for Jetson

This is a perfect fit for Jetson Thor (128GB) and AGX Orin 64GB — the memory and bandwidth make 27B dense at Q4–Q8 with speculative/MTP decoding a headline use case. Jetson AI Lab already has a Qwen3.6 27B model card but does not cover the MTP draft-model flow, which is what's unlocking 2.5x throughput and making agentic coding usable locally.

Suggested tutorial scope

• Build llama.cpp with the MTP PR (#22673) on JetPack 7.x for Thor and JetPack 6.x for AGX Orin
• Run Qwen3.6-27B Q4_K_XL / Q5_K_XL / Q8 with MTP draft head; measure tok/s on Thor, AGX Orin 64GB, and Orin NX 16GB (where feasible)
• Compare against non-MTP baseline and against Gemma 4 31B MTP for agentic coding
• Wire up drop-in OpenAI / Anthropic API endpoints so users can plug it into OpenCode, Aider, Continue.dev
• Report 262k-context memory footprint, prefill latency, and slot-reuse tricks (see --slots trick from the Ralph-loop post)
• Include quality-vs-quant comparison aligned with the community's BF16/Q8/Q6/Q4/IQ4/IQ3 matrix

---

Filed by JetsonPulse

[shahizat]

+++, MTP support for the main brash just landed: ggml-org/llama.cpp#22673.

Also, it would be quite interesting to have a tutorial on using DFlash(https://github.com/z-lab/dflash) as well.

[LiangSu8899]

Qwen3.6-27B on Jetson AGX Thor: up to ~53 tok/s at 16K context

Quick share for anyone tracking large-model deployment on Jetson AGX Thor.

I brought up Qwen3.6-27B NVFP4 on a single Jetson AGX Thor using FlashRT.

• Hardware: Jetson AGX Thor, SM110, 128GB LPDDR5X
• Model: Qwen3.6-27B NVFP4
• Runtime: FlashRT CUDA / CUTLASS kernel pipeline
• No ONNX export
• No TensorRT engine compile
• No per-driver rebuild
• Static CUDA Graph execution
• MTP speculative decoding with paired FP8 MTP checkpoint
• FP8-KV XQA attention path for long context

The same kernel/runtime path is shared with discrete RTX cards. The Thor path adds a hardware-specific MTP-fc M-tile kernel using 160KB dynamic shared memory, gated on the SM110 opt-in limit.

Results

Warm-state decode, max_new_tokens=64, K=6, repeated text prompt.

Context	TTFT	Decode tok/s	AL	K=1 vs K=6 greedy parity
128	268 ms	42.8	3.86	PASS
2K	3.46 s	42.5	3.71	PASS
8K	9.78 s	52.2	4.33	PASS
16K	19.23 s	52.9	4.82	PASS

K=1 vs K=6 greedy parity is the spec-decode correctness check: under temperature-0 greedy decoding, the unspeculative path and the full K-step speculative chain should produce identical token sequences.

A PASS means the NVFP4 + MTP speculative pipeline introduces no token-level drift relative to the greedy reference in these tests.

These numbers look encouraging to me for a dense 27B-class model on a single edge SoC, but I would really appreciate feedback from the Jetson community on whether the setup / measurement protocol can be improved.

Current direction

I’m still working on the Thor path. The next target is roughly 60–70 tok/s decode on Jetson AGX Thor.

Main directions:

• more aggressive MTP-tail fusion
• tighter NVFP4 GEMM tile scheduling
• further reduction of non-GEMM / QDQ / layout overhead

Repo

https://github.com/LiangSu8899/FlashRT

The Thor path is currently on branch feat/qwen36-thor.

Docs:

• docs/qwen36_nvfp4.md
• docs/qwen36_usage.md

For comparison, the RTX 5090 reference path reaches around:

• ~145 tok/s at 16K context
• ~158 tok/s at 128K context

The OpenAI-compatible server also auto-dispatches the Thor frontend on SM110:

• examples/qwen36_openai_server.py

So the runtime surface is the same as the RTX flow.

I’m sharing this here because I think Qwen3.6-27B on Thor is an interesting edge-deployment case, and I’d be very interested to hear thoughts from the Jetson community.

If there are better ways to benchmark this, or if anyone has suggestions on Thor-specific tuning, I’d really appreciate the feedback.