MoE decode carries ~5-6.6 ms/step of fixed cost that dense models do not (Arc Pro B70 / BMG-G31)

[doopeworld]

On Intel Arc Pro B70 (BMG-G31, 608 GB/s), single-request decode of Qwen3.5-MoE hybrid models is limited by a fixed per-step cost of roughly 5-6.6 ms that is independent of how many bytes the model reads. A dense model of the same architecture family, on the same card and the same build, shows no such fixed cost and runs at the memory-bandwidth roofline.

Removing this overhead would take these models from ~71-89 tok/s to ~118-217 tok/s at batch 1.

Environment

  |   -- | -- GPU | Intel Arc Pro B70 32GB (BMG-G31), 608 GB/s CPU / RAM | Core Ultra 9 285H, 96 GB DDR5-5600 vLLM | 0.26.1.dev0+g568afb3a1 vllm-xpu-kernels | 0.1.12.1 torch | 2.13.0+xpu Env | VLLM_XPU_ENABLE_XPU_GRAPH=1, VLLM_USE_V2_MODEL_RUNNER=1, async scheduling on Models | Qwen3.5-MoE hybrid (40 layers: 30 GatedDeltaNet + 10 full attn, 256 experts, top-8 + 1 shared, moe_intermediate 512, hidden 2048)

8x the work for 1.21x the time. Consistent with a large fixed per-step cost that amortizes across the batch.

What was ruled out

Each of these was measured, not assumed.

Quantization format / coverage. MXFP4, GPTQ-Int4 and AutoRound-int4 all land in the 71-89 tok/s band despite an 85% spread in bytes read.

Level Zero runtime configuration. 14 configurations swept — UR_L0_USE_IMMEDIATE_COMMANDLISTS=0, UR_L0_BATCH_SIZE 16/64/128, UR_L0_REUSE_DISCARDED_EVENTS, UR_L0_DISABLE_EVENTS_CACHING,UR_L0_SERIALIZE, UR_L0_IN_ORDER_BARRIER_BY_SIGNAL, legacy SYCL_PI_* equivalents, and both Level Zero adapter versions. All results 12.56-12.67 ms except SYCL_UR_USE_LEVEL_ZERO_V2=0 at 13.33 ms. No knob moved it.

Kernel launch batching. XPU graphs work correctly. 200 sequential torch.relu calls on a 256-element tensor:

eager   0.862 ms   (4.3 us per launch)
replay  0.181 ms   (0.9 us per launch)   -> 4.75x

Graph capture also demonstrably matters end to end: enforce_eager=True gives 41.09 ms/step vs 12.6 ms with graphs.

vLLM Python-side work. In-process cProfile (VLLM_ENABLE_V1_MULTIPROCESSING=0), 64 decode steps, sorted by tottime:

63 calls  0.618 s   torch/xpu/graphs.py:105(replay)
64 calls  0.090 s   Event.synchronize
   ...
64 calls  0.002 s   gpu/model_runner.py:874(prepare_inputs)
64 calls  0.002 s   gpu/sample/sampler.py:70(__call__)
64 calls  0.002 s   gpu/attn_utils.py:566(build_attn_metadata)
65 calls  0.001 s   v1/core/sched/scheduler.py:425(schedule)

Scheduler, input preparation, attention metadata and sampling together account for roughly 0.15 ms/step. vLLM's Python layer is not the cost.

Empty expert group dispatch. XpuFusedMoe._apply_kernel calls cutlass_grouped_gemm_interface with num_experts=self.num_experts (256) on every token, though only 9 experts have rows. In MoEGEMM(grouped_gemm_xe2.hpp) every work-group serially scans all 256 entries of rows_per_expert to build a prefix sum before locating its tile.

An isolated microbenchmark suggested this was expensive (232 us for 256 groups vs 94 us for 8, identical work). It is not the cause. We patched cutlass_grouped_gemm_interface to accept an optional expert_ids tensor so group ireads ptr_B[expert_ids[i]], allowing the caller to pass only active experts with no weight copying, and verified the compaction engages (8 groups instead of 256). End-to-end gain at batch 1: zero (72.6 vs 74.5 tok/s, within noise).

Reporting it anyway since the 256-group dispatch is real and may matter at other shapes — happy to open a separate PR if useful.

Where the time appears

All remaining time is inside torch.xpu.XPUGraph.replay() — 9.8 ms/step by tottime. Since replay() blocks until completion, this is submit-plus-wait and cannot by itself distinguish genuine GPU execution from replay overhead.

Note that torch.profiler reports only 2.38 ms/step of XPU kernel time (152.4 ms over 64 steps), but it does not appear to capture kernels executed inside graph replays, so we do not rely on that figure.

The fixed-cost fit above is independent of any profiler and holds across four models and three quantization formats.

What would help

1. Confirmation of whether ~5-6.6 ms/step of MoE-specific fixed cost is expected on this path, or a known gap.
2. Guidance on measuring GPU time inside XPU graph replays — the profiler undercount makes attribution difficult for anyone hitting this.
3. If the 256-group grouped-GEMM dispatch is considered worth fixing regardless, we have a working expert_idspatch against v0.1.12.1.

Reproduction

bash

# byte counts from safetensors headers + config
# batch sweep, prefill subtracted, median of 3
BATCHES=1,2,4,8 NGEN=64 python3 batchsweep.py /models/<model>

Scripts used for all measurements above can be attached on request.

[doopeworld]

Update: implemented sparse expert dispatch — added an optional expert_ids tensor to cutlass_grouped_gemm_interface so group i reads ptr_B[expert_ids[i]], letting the caller pass only active experts (8 instead of 256) with no weight copying.
Result at batch 1: 70.86 → 74.55 tok/s (+5.2%). Real but far short of what the isolated microbenchmark predicted (232 µs → 94 µs for the same work). The empty-group scheduling is genuinely wasteful, but it is not the dominant cost in the ~5-6.6 ms fixed residual.
The residual therefore remains unexplained. Ruled out so far: quantization format, Level Zero runtime configuration (14 configs), graph capture batching (verified working, 4.75× on a microbenchmark), vLLM Python-side work (~0.15 ms/step by cProfile), empty expert group dispatch, and workspace allocation reuse (see #392).

[doopeworld]

Follow-up: expert-count A/B rules out gathered bytes as the decode bottleneck

Additional measurement on the fixed per-step cost reported above. Both arms run back-to-back in a single session, same container, same flags, only num_experts_per_tok changed in config.json.

Setup

• 1× Intel Arc Pro B70 32GB (BMG-G31, Xe2), ZE_AFFINITY_MASK=1, 275W cap
• vLLM v0.26.0 (commit 568afb3a1, reports 0.26.1.dev0), torch 2.13.0+xpu
• VLLM_XPU_ENABLE_XPU_GRAPH=1, VLLM_USE_V2_MODEL_RUNNER=1
• Model: Qwen3.6-35B-A3B-GPTQ-Int4 — 40 layers, 256 experts, top-8, hidden 2048, moe_intermediate 512
• Server args identical between arms, no --speculative-config: --dtype float16 --max-model-len 32768 --max-num-seqs 16 --kv-cache-dtype auto --max-num-batched-tokens 16384 --block-size 64 --language-model-only --trust-remote-code
• Benchmark: llama-benchy 0.4.0, --skip-coherence --exact-tg --tg 128 --pp 4096

Results

num_experts_per_tok | decode tg128 (t/s) | prefill pp4096 (t/s) -- | -- | -- 8 (stock) | 76.24 ± 0.03 | 10,911 ± 54 2 | 81.68 ± 0.82 | 14,924 ± 49 ratio | 1.07× | 1.37×

num_experts was left at 256 in both arms — all experts stay resident, VRAM footprint and kernel count are unchanged. The only variable is how many expert weight matrices are gathered per token, which drops ~4×. Output at top-2 is incoherent (expected; coherence check disabled deliberately, timing only).

Interpretation

Cutting MoE weight traffic ~4× yields 1.37× on prefill but only 1.07× on decode. Prefill scales with expert work roughly as expected. Decode is nearly insensitive to it.

The prefill arm doubles as an internal control: same card, same session, same PCIe link, same memory subsystem, same kernels. Since prefill responds normally, the decode residual cannot be attributed to the device, the link, or achievable memory bandwidth — it is specific to the decode path.

Combined with earlier results

The fixed per-step cost is now measured to be invariant to:

• quantization format (MXFP4 / GPTQ-int4 / AutoRound-int4 all ~74 t/s)
• gathered bytes per token (this test)
• expert-group dispatch density (256→8 group kernel patch: +5%)
• Level Zero runtime configuration (14 configs swept)
• MoE workspace allocation churn (PR [MoE] Allow reusable workspaces in XPU fused MoE #392 approach: no effect at batch 1)

And it is:

• absent in a dense hybrid model on the same setup (0.00ms fitted residual)
• present at 4.9–6.6ms in all three MoE models tested
• host-bound in signature on a batch sweep (8× work → 1.2× time)
• only 17% smaller under OpenVINO 2026.3 on the same model and card (87 vs 74 t/s decode), i.e. not vLLM-scheduler-specific

Known caveat

The host shows a continuous pcieport 0000:01:00.0: RxErr (First) flood and a second B70 that intermittently drops off the Level Zero bus. The A/B above was run on the stable card with both arms in the same session, and the prefill control argues against link error rate as the cause — but flagging it for completeness.

Ask

Is there per-decode-step work in the XPU MoE path that is independent of both expert count and gathered bytes — e.g. per-layer synchronization, routing/scatter bookkeeping, or host-visible state updated inside XPUGraph.replay()? Profiling from this side cannot see inside graph replay, which is where the residual appears to live.

[jikunshang]

MoE have a lot of small ops, which will cause significant kernel launch overhead in small batch size senario. Ideally, Graph could help improve performance. Unfortunately, current XPU Graph have some limitation, graph.reply() performance is not what you expected(eliminate all most host overhead). You can expect torch 2.14 could improve this.

[doopeworld]

Thank you for the input, Waiting for it then !!! Envoyé de mon iPhoneLe 9 août 2026 à 05:33, Kunshang Ji ***@***.***> a écrit :jikunshang left a comment (vllm-project/vllm-xpu-kernels#509) MoE have a lot of small ops, which will cause significant kernel launch overhead in small batch size senario. Ideally, Graph could help improve performance. Unfortunately, current XPU Graph have some limitation, graph.reply() performance is not what you expected(eliminate all most host overhead). You can expect torch 2.14 could improve this. —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[AlexanderScha]

Second data point for this, from Arc Pro B60 (BMG-G21) on Intel's fork rather than the upstream stack — the dense/MoE split you describe reproduces cleanly, and the size of it may be useful.

intel/llm-scaler-vllm:0.21.0-b3, TP2 for the dense model and TP4 for the MoE, identical settings on both sides except --enforce-eager removed and VLLM_XPU_ENABLE_XPU_GRAPH=1 added. Plateau of 3 runs, first discarded.

model		eager	graphs	Δ
Qwen3.6-27B-FP8 (dense)	single stream	22.42	22.26	−0.7 %
	N=8	147.3	151.9	+3 %
Qwen3.6-35B-A3B (MoE)	single stream	56.51	83.43	+48 %
	N=8	290.5	494.3	+70 %

So on this hardware the dense model gains nothing measurable and the MoE gains roughly half again — consistent with "dense runs at the roofline, MoE carries a fixed per-step cost that graphs amortise".

One practical detail that may be worth adding to the issue, since it stops the server from starting at all: with the default capture list [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64], capture dies on the fifth shape (size 16):

RuntimeError: level_zero backend failed with error: 40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)
  torch/xpu/graphs.py:85, in capture_begin

It is not memory pressure — the failure lands on the same shape at --gpu-memory-utilization 0.85 and 0.70, and it happens in capture_begin, before any execution. --compilation-config '{"cudagraph_capture_sizes":[1,2,4,8]}' gets past it, and the numbers above are with that list. Larger --max-num-batched-tokens makes it worse (40960 fails where 8192 succeeds).

Note this is torch 2.13 inside Intel's fork image, not your upstream build, so it may be a fork-specific limit rather than a torch-xpu one. If it is not, it would cap the benefit at small batches for anyone hitting it.

Also visible in the log, and possibly relevant to how much is left on the table:

FMHA sycl-tla kernels cannot be captured with XPU graphs,
falling back to PIECEWISE graph mode on XPU platform.

Attention is outside the graph, so +48 % is what the surrounding pieces alone are worth.

Full write-up with the method: https://github.com/AlexanderScha/battlemage-vllm/blob/main/measurements/xpu-graphs-b60.md