Speculative Decoding Reduces tok/sec Speed on M1 Pro Mac

[ibehnam]

Using LMStudio client, I've tried the following combinations:

• Llama 3.1 8B Q4 MLX (main) + Llama 3.2 1B Q4 MLX (draft): speed went from 38 t/s (without draft) to 33.93 (with draft)!
• Same thing except I used Llama 3.2 3B (smaller gap with the main model) Q4 MLX: went from 38.7 t/s to 29.29 t/s
• Qwen 2.5 7B Q4 MLX (main) + Qwen 2.5 1B Q4 MLX (draft): 37.08 t/s to 22.54 t/s

MacBook Pro with M1 Pro Chip, 32GB unified memory. I replicated the same results with GGUF models as well.

[JJJ-M]

Hi,
Same story here.
Mac Studio M1 Ultra, 128 GB unified memory.
All models in 8bits.
Models fit in VRAM, no Swap.
<50% draft tokens accepted.

• Deepseek-r1-distill-qwen-32b-mlx (main) + Deepseek-r1-distill-qwen-1.5b-mlx (draft): 14.5 with draft / 16.5 without.
• deepseek-r1-distill-llama-70b (main) + deepseek-r1-distill-llama-8b (draft): 7.3 with draft / 7.9 without.

I tried with Deepseek-r1-distill-qwen-1.5b-mlx (draft) in 4 bits --> No difference with 8bits

[yagil]

Related:

• Poor Speculative Decoding Performance on M2 Ultra ml-explore/mlx-examples#1281

[neilmehta24]

Seeing another report of this behavior here #113 . Fixing this issue is blocked until ml-explore/mlx-examples#1281 is resolved.

[async0x42]

Same issue here with M4 Max

[PaulShiLi]

Seems like issue is still here with the M2 Max