[WIP] Blog: announcing vllm-metal - #314
Draft
WindChimeRan wants to merge 16 commits into
Draft
Conversation
ericcurtin
approved these changes
Aug 27, 2026
LxYuan0420
reviewed
Aug 27, 2026
|
|
||
| ## Reproducing the benchmarks | ||
|
|
||
| The cross-engine serving benchmarks use the SiliconBench agent split: 100 prompts averaging 4.6K input and 70 output tokens, run closed loop at fixed concurrency on an Apple M5 Pro with 64 GB running macOS 26.6. The NAX A/B instead uses the two Sonnet configurations described above: 100 prompts at request rate 10 and concurrency 32. |
There was a problem hiding this comment.
Quick clarification: I couldn’t find the two Sonnet configurations mentioned here. Were they described elsewhere?
ericcurtin
reviewed
Aug 27, 2026
Contributor
There was a problem hiding this comment.
Nice writeup, the packed-varlen vs padded framing is the strongest part. arch.png, gemma-e4b.png, kv-layout.png and spark-vs-apple.png are unreferenced, since the post links the .svg versions. Only siliconbench-27b.png is needed, for social_image. Drop the other four.
|
|
||
| ## Reproducing the benchmarks | ||
|
|
||
| The cross-engine serving benchmarks use the SiliconBench agent split: 100 prompts averaging 4.6K input and 70 output tokens, run closed loop at fixed concurrency on an Apple M5 Pro with 64 GB running macOS 26.6. The NAX A/B instead uses the two Sonnet configurations described above: 100 prompts at request rate 10 and concurrency 32. |
Contributor
There was a problem hiding this comment.
"the two Sonnet configurations described above": Sonnet isn't mentioned anywhere above. The NAX section calls them the prefill-heavy and standard splits. Pick one name and use it in both places.
|
|
||
| The DGX Spark comparison reuses that split unchanged, on a GB10 box with 121 GB of usable unified memory running the same harness and the same client. | ||
|
|
||
| Stats cover completed requests; an empty response counts as failed. The harness and per-engine configurations live in the [SiliconBench repo](https://github.com/WindChimeRan/SiliconBench). The MTP arms ran vllm-metal `0.3.0.dev20260821152549` with the serve command from the quickstart. |
Contributor
There was a problem hiding this comment.
The post announces v0.4.0 but the MTP numbers come from a 0.3.0.dev build. Either re-run against the release wheel or say plainly in the text that the MTP arm predates it.
|
|
||
| On M5 Macs, vllm-metal automatically uses the NAX kernel for compatible prefill batches; no extra configuration is required. Pre-M5 Macs keep using the existing path. | ||
|
|
||
|  |
Contributor
There was a problem hiding this comment.
This is the only figure with no .svg and no generator script. Every other chart is reproducible from make_fig*.py; add one for consistency.
|
|
||
| The same packed batch structure also supports batched speculative decoding. vllm-metal's Gemma 4 MTP assistant drafts one token per request per step, and the draft pass itself is batched: every request's seed runs in one grouped varlen dispatch against the target's shared KV cache. Verification runs in the next packed target batch. Speculation never leaves the continuous-batching path, so it can be measured on the same agent split as everything above; the dashed blue line in the Gemma 4 figure is this arm. Change against the vllm-metal baseline: | ||
|
|
||
| | c | Wall | Output tok/s | ITL avg | TTFT avg | Acceptance | |
Contributor
There was a problem hiding this comment.
Nit: spell out c as Concurrency, and state in the header that the deltas are vs. the no-MTP baseline.
WindChimeRan
force-pushed
the
vllm-metal-v0.4.0-blog
branch
from
August 29, 2026 23:11
d3c13d0 to
bf52453
Compare
Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
WindChimeRan
force-pushed
the
vllm-metal-v0.4.0-blog
branch
from
August 30, 2026 03:04
bf52453 to
2d9a70b
Compare
The cross-machine comparison answered a question a vllm-metal reader cannot act on, and reported the gap without the mechanism behind it. It moves to a standalone post where it can be developed properly. Removes the section, its two appendix items, and spark-vs-apple.svg, and leaves a placeholder for an M1 Pro / M5 Pro generational comparison in its place. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
Gemma 4 E4B at concurrency 1 and 8; M5 Pro column filled from the existing agent-split results, M1 Pro placeholders pending a run. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
Upstream cut v0.28.0 on 2026-09-01, not v0.4.0. The release list is v0.1.0, v0.2.0, v0.28.0; the 0.4.0 line existed only as dev tags, and upstream pyproject.toml now reads version = "0.28.0". Per the release notes, v0.28.0 aligns the plugin release number with its vLLM v0.28.0 core dependency and covers 152 merged PRs since v0.2.0. Renames the post and updates the release-name references. Leaves the appendix sentence about the MTP measurement vintage alone: that is data-dependent and belongs with the figure regeneration. The asset directory keeps its 2026-08-25-vllm-metal-v0-4-0 path, which is invisible to readers and would otherwise churn the figure scripts. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
Re-measures the whole agent-split matrix at 4-bit, which is what most Mac users deploy, on a uniformly upgraded stack: vllm-metal 0.28.0.dev20260901062632 against vLLM 0.28.0, llama.cpp 0eadefeb, oMLX dc312e6e. Adds a mixture-of-experts arm, Qwen3.6-35B-A3B, so the padding argument's claim about wasted rows in "attention, MLP, and MoE" has a measurement behind its last term. Every concurrency level runs against its own freshly started server, and oMLX additionally gets an empty cache directory each time because its prefix cache lives on disk. Levels sharing a server also share its caches: on this split, whose prompts repeat, that is worth 1.67x throughput and 4.1x TTFT at concurrency 8, which is enough to change which engine appears to win. Qwen3.8-27B: vllm-metal holds the flattest TTFT curve, 7.3 to 9.3 s across a fourfold concurrency increase, against 11.4 to 15.7 for llama.cpp and roughly 38 for both oMLX modes; it also has the lowest end-to-end latency and is the only engine whose output throughput rises with concurrency. The two oMLX modes converge once neither can carry a warm cache between levels. Gemma 4 E4B: 2.3 s TTFT at concurrency 16 against llama.cpp's 32.4 s and oMLX's 22.9 s. The MTP arm takes the top throughput line at 79 tok/s. Batched MTP: regenerated delta table. MTP pays most single-stream (+20% output at concurrency 1), returns 9% at concurrency 16, and is a wash at 8. Adds the draft-depth result: acceptance falls 71% to 62% to 52% from one to three draft tokens while throughput stays flat, so one draft token is the recommendation on grounds of simplicity. Figures are deployment comparisons: each engine runs the 4-bit build its own users would install and at its own memory default, so weight footprints are not matched across engines. The appendix states both. Drops mlx_lm from the figures - it completes 23-57% of requests through tool-call parse truncation, and was already absent from the 27B comparison. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
oMLX reaches first token soonest single-stream on the 27B and is slightly ahead on latency and throughput there too. The figure shows it; the prose skipped it and went straight to the curve shape. Say it, and say why the ordering inverts under load. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
The column still held the BF16 numbers and contradicted the Gemma figure in the same post. The M1 Pro column remains unmeasured. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
The post argued that packing saves padded rows in "attention, MLP, and MoE" while every benchmarked model was dense on that last term. This arm supplies it: 35B total parameters with 3B active per token, experts holding 93% of the checkpoint's text weight, and gated-delta-net linear attention alternating with standard attention, so one model exercises both the hybrid path and the expert path. vllm-metal holds TTFT nearly flat at 1.6 to 2.0 s from concurrency 1 to 4 where both oMLX modes climb to about 6.5 s, and reaches 35.3 output tok/s against 30.1 and llama.cpp's 25.3. oMLX is marginally ahead single-stream on latency and throughput, as it is on the 27B. Also names the model in the supported-models bullet, which previously listed only the 27B. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
The second oMLX series was measured with --no-cache and labelled "bounded in-memory mode". It had no prefix cache at all: oMLX honours --hot-cache-max-size only when a cache directory is set and forces it to 0 otherwise, so the 8GB passed alongside --no-cache never applied. The figures drew that arm as plain "oMLX", so the line a reader would take for the default was the one with caching switched off. Both oMLX arms are now named for where the cache lives. "oMLX (SSD offload)" is unchanged and is oMLX's own default: disk-backed, memory tier off, its 100 GB cap being what "auto" resolves to on a 1 TB machine. "oMLX (RAM cache)" is new, holding the prefix cache in memory and writing nothing to disk, which is what llama.cpp and vllm-metal do — the comparable configuration. Where the cache lives turns out to matter more than whether it exists, and the two modes no longer coincide, so the paragraph explaining why they overlapped is gone. On the 27B, in memory against on disk is 5.47 against 4.80 tok/s and 32.9 against 38.9 s TTFT at concurrency 4. On the MoE it is 33.9 against 30.1 tok/s, which also moves that model's story: oMLX now leads single-stream throughput 33.7 to 28.3 and sits a tenth of a second from vllm-metal on end-to-end latency, so vllm-metal's advantage there is time to first token rather than throughput. Gemma gains the same pair, 14.1 s against 22.9 s TTFT at concurrency 16. The MoE figure also gains mlx_lm, drawn in grey and labelled with its completion count: it returns an empty completion for roughly six in ten of these long-input prompts, so its rates cover only the requests that finished. Every arm was re-measured with one server per concurrency level, so no level inherits a warm cache from the one before it. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
The reproduction block still described one oMLX arm by the flags of the old no-cache run and referred to a "bounded-memory arm" that no longer exists. It now documents both arms as they were actually served: the SSD arm, which is oMLX's default apart from the fresh cache directory, and the RAM arm, which adds OMLX_HOT_CACHE_ONLY and a hot-cache size so the prefix cache stays in memory and the directory stays empty. "Single-stream, oMLX is ahead" was true of the arm the post used to draw and is still true of the SSD arm, but not of the RAM arm, which trails vllm-metal at concurrency 1. Attributed to the arm it describes. "The only engine whose output throughput rises with concurrency" was written when the alternatives were flat or falling. The RAM arm ends above where it starts, as does llama.cpp, so the distinction is now that vllm-metal is the only one rising at every step: 5.8 to 6.8 to 7.3 tok/s. Checked against all five arms in that tree. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
The bullets used "in memory" and "with SSD offload" as shorthand that was only defined further up, in a paragraph carrying other setup detail. A bullet now opens the list and says plainly what the two arms are, so the shorthand after it is anchored. The serving-configuration list becomes the commands actually used, and two of its entries were wrong for these runs: -c 49152 (12,288 per slot) -> -c 65536. Every 4-bit arm recorded LLAMACPP_CTX_SIZE=65536, so the slots hold 16,384 tokens, not 12,288. VLLM_METAL_MEMORY_FRACTION=0.7, "the default fraction of 0.5" -> removed. Every vllm-metal arm in these trees recorded FRACTION=auto, and auto has been the default since the 0.5 pin was dropped. The appendix already said this correctly one paragraph earlier, so the post contradicted itself. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
The post put mlx-community/Qwen3.8-27B-4bit at 14.1 GiB of text weights against llama.cpp's 15.3 GiB UD-Q4_K_M. Measured on disk, the checkpoint's safetensors total 15.0 GiB. The disclosure still holds — the footprints are not matched — but the gap is 2%, not 8%. Everything else numeric in the sections these re-measurements touched was checked against the result files and left alone: the MTP delta table (all nine values within rounding), the M1-to-M5 table's M5 Pro column, Gemma's 32.4 s and 2.3 s TTFT at concurrency 16, and the agent split's 100 requests averaging 4.6K input tokens. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
The 16-slot sensitivity table justifies the --parallel 4 setting used in the Gemma figure, but the figure is the 4-bit model and the table is not: its --parallel 4 column matches the BF16 trees exactly — chat 22.4/77.0/81.5 and agent 18.3/44.2/40.2 tok/s in results/Gemma-4-E4B-it/m5pro/. Naming the model so a reader is not left to assume the two share a checkpoint. Also checked and left alone, both correct as written: the MoE's experts hold 92.9% of the checkpoint's text weight (93.2% counting the shared expert) against the stated 93%, computed from the safetensors headers with the vision tower excluded, which is what "text weight" is doing in that sentence; and the six NAX percentages, which match PR_nax_prefill_draft.md in the vllm-metal repo to the rounding shown. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
The post gave 73.0, 73.7 and 72.6 tok/s at concurrency 8 for draft depths 1, 2 and 3. No result file in the repo held any of those: the depth-2 and depth-3 servers started on 2026-09-01, per their serve logs, but no benchmark JSON was ever written for either, and the depth-1 arm that is plotted reads 72.9, not 73.0. Re-measured depths 2 and 3 at concurrency 8 with the per-level protocol, on the same build as the depth-1 arm in the Gemma figure. Depths 1, 2, 3 now read 72.9, 74.7 and 73.5 tok/s. The claim is unchanged — throughput is flat across depth, spread 2.5%, and the extra accepted tokens still cancel the extra draft compute — but every number now comes from a file. The acceptance rates in the same sentence (71%, 62%, 52%) are left as they stand. They are not recorded in the result files or the serve logs, and they describe the drafter rather than anything these re-measurements changed. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
The section claimed a 30,000 + 5,000 + 10 batch sends 90,000 rows through the model against 35,010 packed. The arithmetic is right about one padded forward, but it is not what a server does with that mix: a prompt long enough to make padding expensive holds the model thread for its whole prefill and finishes before anything else is admitted. Four send orders against mlx_lm never produced the padded batch. Replaced with a measurement that isolates the cost. Two batches of three requests carry the same total work, about 5,950 prompt tokens; only the longest prompt changes, so padded rows go 5,973 -> 11,970 while real rows stay flat. mlx_lm slows by 72% and nothing else moves: vllm-metal +5%, llama.cpp 0%, oMLX -10%. Solving mlx_lm's two numbers gives prefill 2.40 s and decode 0.92 s, so the prefill doubles exactly as B x max_length predicts and the decode does not, which is why wall time moves 72% rather than 100%. TTFT spread confirms the batches formed: 0.03 s or less for mlx_lm in both, over 1.5 s for oMLX. oMLX was listed as a padding engine with `[B, T_max]` queries. It borrows mlx_lm's BatchGenerator but constructs it with `prefill_batch_size=1` (scheduler.py), so it never pads a prefill, only batches decode. Its row now says so, and the column is renamed from "Encoding" to "Prefill batching", which is what it was describing. The stack count in the lead-in was wrong at three and is now four. The figure is redrawn to the same shape as the table, 4,000 + 1,000 + 1,000, and its layout offsets now scale with the axis instead of being tuned for 30,000, which made the labels collide at the smaller size. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
Three requests understated the cost. mlx_lm pads to `B x max_length` with a prefill batch width of 8, so a 3-wide batch is well short of what the engine will actually form. Re-ran the same A/B at eight requests, which is exactly that width: same ~5,980 prompt tokens either way, padded rows 5,992 against 23,856. mlx_lm slows by 143%, up from 72% at three requests, so the cost tracks batch width as the shape predicts. Its two numbers give 2.16 s of prefill and 2.36 s of decode; batch B pads the prefill fourfold and leaves decode alone. Everything else lands between -6% and +1%. Two runs per cell, median reported, fresh server each time so no cell inherits a warm prefix cache. TTFT spread separates the engines that batched from those that did not: 0.05 s for mlx_lm, 0.17 s for llama.cpp, 3.25 s for oMLX. Two results flipped at the wider batch and are now stated. vllm-metal is fastest in both batches and the only engine faster on B than on A; at three requests mlx_lm's padded batch beat it on the uniform set. And lily, fastest of all at three requests, drops behind vllm-metal at eight, because eight requests contain overlap it cannot take. lily is added as a column with a link to its repository, since it is new, and with its constraints stated: batch size 1, no concurrency, no streaming, and it loads only Qwen3.6-35B-A3B-4bit. It earns a place as the clean control — an engine that never batches is perfectly insensitive to the length distribution, 1.01x across the two batches. The figure is unchanged. It shows the mechanism with three sequences, which is clearer than eight, and no text ties its numbers to the table's. Signed-off-by: Ranran Haoran Zhang <ranranhaoranzhang@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a post announcing vllm-metal on Apple Silicon with MLX backend
Comparison with DGX-Spark is WIP.
Feedback on the technical framing and benchmark methodology is welcome now.
@LxYuan0420 @ricky-chaoju @ericcurtin @mgoin