Reliable GLM-5.2-oQ4e-mtp on one M3 Ultra 512GB (v0.5.0): config recipe, SSD prefix cache 40×, MTP is a win, and the real long-context ceiling

[guruswami-ai]

Working recipe + findings: reliable GLM-5.2-oQ4e-mtp on a single M3 Ultra 512GB (v0.5.0)

Sharing a config recipe and a few measurements from getting Jundot/GLM-5.2-oQ4e-mtp (423.8 GiB)
running reliably on one Mac Studio M3 Ultra 512GB, macOS 26.5.2, omlx v0.5.0 (source build,
custom kernels, mlx 0.32.0). Three things had to be right before it was stable, then the SSD prefix
cache and MTP both paid off well. Posting in case it saves someone the two days it cost us.

The config that made it reliable

1. MLX_METAL_FAST_SYNCH=0. With it set to 1, every GLM generation hung forever on the first
   forward pass — GPU pinned at 100% residency but only ~4W (a spin-wait, not compute), no token, no
   error, and the executor thread wedged so all later requests queued forever. =0 fixed it
   completely with no measurable perf cost at our scale. (Filed as GLM-5.2 generation hangs forever (silent GPU spin-wait) with MLX_METAL_FAST_SYNCH=1 on M3 Ultra — FAST_SYNCH=0 fixes #2183 — bare mlx_lm.generate
   with the same model/kernels never hung; the trigger needs the server's stream usage + FAST_SYNCH.)

2. iogpu.wired_limit_mb=499712 (488 GiB), not 510 GiB. The enforcer log suggests raising toward
   522240; on a 512 GiB box with this 424 GiB model that pushed the OS into jetsam, which killed the
   server mid-load, which (with launchd KeepAlive) became a compounding crash-loop. 488 GiB leaves the
   424 GiB model + ~60 GiB KV with the OS safe. (Filed as Hard-killed process holding a large wired model leaks the wired memory at kernel level (reboot-only recovery); KeepAlive turns it into a jetsam crash-loop #2184.)

3. Graceful stops only + generous ExitTimeOut. A hard-killed process (SIGKILL/jetsam) holding
   ~424 GiB wired leaks the whole allocation at kernel level — it survives process death and only a
   reboot reclaims it. Graceful bootout (SIGTERM) released everything (wired 424 GiB → 5 GiB in
   seconds). So never kickstart -k with a big model loaded. (Also Hard-killed process holding a large wired model leaks the wired memory at kernel level (reboot-only recovery); KeepAlive turns it into a jetsam crash-loop #2184.)

Plus the expected: source-built kernels (OMLX_WITH_CUSTOM_KERNEL=1, mlx 0.32.0 + nanobind 2.13.0),
--memory-guard-gb 488, small --hot-cache-max-size for a single-pinned-model node.

SSD prefix cache — the payoff (this is the recipe worth copying)

--paged-ssd-cache-dir <dir> --paged-ssd-cache-max-size 600GB, plus cherry-picking d5fcb22a
(#2180) onto the v0.5.0 tag. Needle test at 131k tokens:

Query	prompt_tokens	cached_tokens	wall time
1st (cold prefill)	147,691	0	1385 s (~23 min)
2nd–6th (same prefix)	147,691	147,456	20–42 s

~40× faster on any resumed/iterated long-context session. The #2180 cherry-pick is load-bearing:
without it, an adaptive-prefill/memory-guard event discards the reconstructed SSD prefix and cold-
prefills the whole prompt — exactly the case that fires on a node sitting near the memory ceiling.
Would suggest it belongs in the next tag.

MTP is a net win on GLM-5.2 post-#2160 (counter to the fine-grained-MoE caution)

The common guidance is that MTP is a net loss on fine-grained (8/256) MoE because verify batches load
near-disjoint expert unions (#2077 etc.). With Lightning MTP (#2113) + the #2160 verify-routing fix
on the oQ4e quant, we measured the opposite — MTP on vs off (llama-benchy, greedy):

Context	MTP on	MTP off	Δ
short (d0)	19.2 tok/s	15.7	+23%
131k depth	13.7 tok/s	12.9	+6%

So on this model+build, keeping MTP on is correct at every context length we tested. Worth noting for
anyone who disabled it based on the general MoE caution.

Retrieval quality + the real long-context ceiling

Needle-in-haystack at 131k: 5/5 needles at 10/30/50/70/90% depth, plus a full-recall sweep, all
correct, and every re-query hit the SSD prefix cache (~20–40 s vs the 23 min cold). No sign of the

128k top-k degradation reported elsewhere, at least at 131k on oQ4e.

Pushing higher, the binding limit turned out to be prefill peak memory, not retrieval quality and
not steady-state KV. At 196k the one query that ran returned the needle correctly (quality holds),
but subsequent queries were rejected by the prefill memory guard: prefill would require ~494.81 GB peak. 246k was rejected outright (~511.54 GB peak). So on a single 512GB node with this 424 GB
model, the reliable context ceiling is ~131k — prefill transient memory (~495 GB at 196k) hits the
wall long before the steady-state KV would (only ~20 GiB at 196k). Worth flagging because the
max_context_window you can set (we tried 262144) is well above what the node can actually prefill
— it fails safe with a 400, but the practical ceiling is a good deal lower than the configurable one.
(Likely tunable somewhat via smaller prefill chunk size / guard settings — not yet explored.)

Benchmarks vs the v0.5.0 release claims (M3 Ultra, this quant)

• Decode ~19.2–20 tok/s beats the claimed 17.9 (with MTP + GLM-5.2-oQ4e-mtp on v0.5.0rc1 appears slower than GLM-5.2-oQ4 on v0.4.5.dev1 in long-context single-request TG #2160).
• 16k prefill matches (~173 vs 178.9 tok/s claimed).
• 32k prefill below the headline (~113 vs 174.4) — partly oQ4e (~4.6 bpw) vs the release's mxfp4 (~4.25).

Happy to share full llama-benchy JSON / py-spy stacks / logs if useful for any of the linked issues.

[jundot]

Great writeup, thanks. A few updates: the FAST_SYNCH=1 hang was an omlx bug, fixed in 69ea88d (#2183), so it no longer needs to be off with that commit. d5fcb22 is already on main. The other findings here also landed: port binds before pinned preload with /health answering 503 while loading, and the wired-limit recommendation is clamped below physical RAM. One extra data point for the recipe: stranded wired memory from a hard kill also poisons Metal sync machine-wide until reboot.

[guruswami-ai]

Verified on our 5× M3 Ultra 512GB fleet — thanks for the fast turnaround.

Cherry-picked 69ea88d9 (#2183/#2197) onto v0.5.0 (+#2180) and flipped MLX_METAL_FAST_SYNCH back to 1 across all five nodes. The GLM-5.2-oQ4e-mtp first-request hang is gone — it generates normally.

We'd also been carrying a local workaround that forced a non-thread-local MLX stream (new_stream instead of new_thread_local_stream); with 69ea88d9 in place we reverted that too and it's stable — consistent with the fix being about keeping prefill chunk views on the per-engine stream.

Confirmed with FAST_SYNCH=1 across model types:

• GLM-5.2-oQ4e-mtp (reasoning) — generates, no hang
• Kimi-K2.7-Code-MLX (coding) — generates, no hang
• pool: gemma-4 / DeepSeek-V4-Flash / gpt-oss-20b — generate, no hang

Also re-confirmed the #2184 behaviour: graceful SIGTERM cleanly released the ~420 GB wired GLM allocation on stop (27.6M → 0.37M wired pages), no stranding. Appreciate the quick fixes.

[noonkho]

Hi, I'm a beginner in LLM inferencing, really thankful and impressive on this oMLX project, and I am also trying to use GLM 5.2 on one M3 Ultra 512GB MacStudio on the oMLX dmg app v0.5.2rc1 version.

I wonder is it true that only Jundot/GLM-5.2-oQ4e-mtp can turn on the MTP feature? (Which one is it? is it the Lightning MTP? or others?)

Since I am running other models at the same time, it seems that the Jundot/GLM-5.2-oQ4e-mtp with 433 GB seems to be too big which I installed the mlx-community/GLM-5.2-mxfp4 model and the performance isn't that great.

Also I notice that most people would turn off the enable thinking when running but then would it be the same as GLM 5.2 (max) which I see on the benchmark leaderboard?

Just clicking the Preset Profiles GLM-5.2 won't set up the configs mentioned above and I wonder how to get similar performance?

---

My current set up is:
GLM-5.2 preset profile (temp:1, top_p: 0.95, enable_thinking is OFF)

For Resource Management:
Memory guard: Aggressive
Hot Cache Limit (In-Memory Cache): 12% (61GB)
Cold Cache Limit (SSD Cache): 20% (744GB)
Max Concurrent Requests: 2
Embedding Batch Size: 32
Chunked Prefill: Off

Cache is Enabled

For generation defaults:
Fallback Context Window: 16384 (How should I set the context window size for GLM 5.2 in general? or have dynamic size? The above mentioned when >= 262144 it fails safe with a 400?)
Max Tokens: 16384
Temperature: 0.7
Top P: 0.95
Top K: 0
Repetition Penality: 1

For Advanced,
Burst Decode: Aggressive (fastest)
Preserve Prefix Cache for Mid-Conversation System Notes is on
Initial Cache Blocks: 1024

---

Inside the app, where can I find the MLX_METAL_FAST_SYNCH to 0/1, the graceful Graceful stops only + generous ExitTimeOut, or the iogpu.wired_limit_mb?

Also this might stupid to ask, in the Performance Benchmark, I saw the Continuous Batching Tests (2x batch, 4x batch, 8x batch), does it mean it will send the pp1024 (input prompt size 1024 tokens) twice, four times, eight times to test the Prefix Hit rate, Memory Hit rate so that from the mentioned above SSD prefix cache — the payoff section it is faster?

If not, how can I increase the batch size so that I can reach that speedup, the high tg TPS shown in the results?

The performance is down below:

Whoever reading this message, thanks for your patience and please kindly tutor me on using this incredible application.