[PrimusInterParis]

TL;DR: When an agent sends a few hundred to a few thousand new tokens, almost every expert kept in RAM is needed, but most of them only for one to three of those tokens. Maya sent every one of them whole over PCIe to the graphics card, about 1 ms each. With MAYA_DS4_PREFILL_CPU=1 the CPU computes the least-used ones itself while the card loads and computes the rest, split per layer so both finish at about the same time. On my PC an agent's turn of 500-700 new tokens is read in 5.4-6.0 s instead of 8.9-9.7 s (4.1-4.9 s together with the DMA of #958). Off by default; the last bits of the answer can move, as they do between the chunked pass and windows.

What

After each layer's routing on the flow (a chunk of at most MAYA_DS4_PREFILL_CPU_MAX_T tokens, default 4096, a layer alone on its device, no helper):

• the CPU-lane experts with rows that were not copied early (flow_spec) are sorted fewest rows first and taken for the CPU while the CPU's estimated time (a row's time, measured once at the load; MAYA_DS4_PREFILL_CPU_ROW_MS sets it) stays under the stream's for the rest (an expert's bytes over that GPU's own link as the load measures it; MAYA_DS4_PREFILL_CPU_EXPERT_MS sets one time for every GPU);
• their virtual slots move behind the streamed ones and the chunk is sorted again (moe_sort), so flow_rest streams only its own and the CPU's rows are one range [pc_r0, pc_r1);
• a host thread runs cpu_experts on them (inputs and row tokens read back to pinned buffers) while flow_layer streams and computes the others;
• pcpu_finish joins it, uploads its rows and adds them with moe_combine (moe_combine_x where the sums are fixed-point) after the layer's streamed and VRAM experts.

cpu_experts now takes a recursive mutex (lane_mu) so the lane's scratch has one caller at a time (the share's thread beside a decode on the other slot); lane_busy= counts the times the share found it taken (0 in every run here). DS4_PREFILL_STATS=1 adds DS4_PREFILL_CPU gpu= layers= experts= rows= cpu_ms= wait_ms= lane_busy= a prompt and GPU. Docs: a section and four rows in docs/SETTINGS.md.

Why

A short prompt on a PC whose experts mostly sit in RAM streams nearly all of them: 600 new tokens touched 7,700 RAM experts (87 GB) on my PC, about 40% of them for one to three rows, at ~1 ms an expert over PCIe 3.0 - while the CPU lane computes such an expert in ~0.5 ms a row. GLM has had the same split since MAYA_GLM_PREFILL_CPU.

Measured

Two Xeon E5-2697 v2 (Ivy Bridge, 12 cores each: AVX and F16C, no AVX2 or FMA), 16x 32 GB DDR3-1600, 2x RTX 3090 24 GB on PCIe 3.0 x16 (both on socket 0); Linux, CUDA 13.0. Maya v1.1.59 (027992c) + this PR (rebased onto main since; only docs changed there), DeepSeek-V4.1-Flash-1017 Maya-S, --tp 2 (falls back to a layer split: one GPU a layer group), --parallel 2, --max-context 1048576, --dspark none, 20 CPU threads; 3,144 experts in VRAM, 12,212 (134.6 GB) copied into RAM in 2 NUMA parts and pinned.

Harness: a fresh engine per run, one frozen copy of expert-profile.bin restored before every start (the profile the server saves at each stop moves the placement otherwise), the same requests every run, MAYA_DS4_SLOT_KEEP=0, MAYA_DS4_PROMPT_PICK=length:96 in every run (so a run's path does not depend on the timing pick). Per run: three fresh prompts (326 / 602 / 1,638 tokens, nothing reused, first token only), then a 20.6K-token start read once and eight reads on top of it (an agent's turn: 500-3,200 new tokens), then the decode bench. Runs in this order: neither setting, DS4_FLOW_COPY_DMA=1 only (#958), MAYA_DS4_PREFILL_CPU=1 only, both, neither again; "without" below is the mean of the two runs with neither.

prompt	without	with (EXPERT_MS 0.9)	change
326 tokens, fresh	7.55 s	4.29 s	-43%
602 tokens, fresh	8.95 s	5.15 s	-42%
1,638 tokens, fresh	11.53 s	8.12 s	-30%
agent's turn: 506-708 new tokens on the 20.6K start (4 reads)	8.90-9.66 s	5.36-6.03 s	-38 to -40%
1,141 / 1,152 new tokens	10.20 / 10.43 s	6.74 / 6.85 s	-34%
3,123 / 3,236 new tokens	12.67 / 12.91 s	9.91 / 10.05 s	-22%
20.6K tokens, cold (chunks of 8,192: no share)	25.49 s	24.76 s	unchanged

With DS4_FLOW_COPY_DMA=1 in both arms (#958, EXPERT_MS 1.2): fresh 6.70 / 7.52 / 9.82 -> 3.18 / 3.92 / 6.59 s (-53 / -48 / -33%), an agent's turn 7.54-8.29 -> 4.08-4.94 s (-40 to -46%), 3,123 / 3,236 new tokens 11.10 / 11.34 -> 8.74 / 9.03 s (-20%). Both against neither: an agent's turn of 500-700 new tokens -49 to -54%.

MAYA_DS4_PREFILL_CPU_EXPERT_MS sweep (a fresh engine each, the same profile), with DS4_FLOW_COPY_DMA=1 on v1.1.56, fresh 326 / 602 / 1,638 tokens - 0.3: 4.5 / 5.3 / 8.1 s, 0.4: 4.2 / 5.0 / 7.7, 0.5: 4.0 / 4.7 / 7.6, 0.9: 3.4 / 4.1 / 6.8, 1.2: 3.2 / 3.9 / 6.6, 1.5: 3.5 / 4.4 / 7.5, 2.0: 4.0 / 5.1 / 8.8 (the CPU the limit from 1.5 on); through the staging 0.9 beat 1.46 (an agent's turn 5.6-6.4 vs 5.9-6.7 s), hence the default 0.9.

Decode (three answers of 300 tokens, greedy, thinking off, two sets of questions A / B): 25.7 / 26.6 tok/s with it, 24.5 / 25.8 and 26.2 / 27.2 without (two runs), 25.7 / 26.2 with both PRs - the same within the runs' spread (the share only runs during prompt chunks).

Answers: the CPU's sums for the CPU's experts, so the last bits move. First-token top-10 distributions of three fresh prompts against the chunked pass without it, KL over the common tokens: 7.1e-4 / 2.1e-3 / 2.1e-3 (326 / 602 / 1,638 tokens) - the same size as the unchanged engine against itself across a restart (1.1e-3 / 5.0e-3 / 8.5e-3) and as windows of 16 against the chunked pass (v1.1.56: 3.8e-4 / 2.1e-3 / 6.0e-2); the first token was the same everywhere.

Update (2026-11-13)

• Since the tests in the comments below: each GPU plans with its own expert time from the link the load measures (152dc6e), and the CPU lane's row time is measured at the load (20e2451), instead of the two fixed values. On the box above they come out at 0.85-1.10 ms an expert and 0.48-0.49 ms a row and plan like the hand-set values (agent turns within about 2% of the hand-tuned 1.2 with DS4_FLOW_COPY_DMA=1); the tables above were measured with the fixed values.
• One GPU (the same box, one RTX 3090, Maya-S, 256K context, --parallel 1, the values measured at the load, two runs per arm): fresh 326 / 602 / 1,638 tokens 8.86 / 10.55 / 13.05 -> 5.11 / 6.51 / 10.05 s (-42 / -38 / -23%), eight agent turns of 506-3,236 new tokens 96.5 -> 66.2 s together (-31%); with DeepSeek: DS4_FLOW_COPY_DMA also where the RAM copies are split across CPU sockets - short prompts 11-16% faster on two sockets, the same bits #958 as well 4.00 / 5.20 / 7.73 s and 56.0 s. Decode the same, the first token the same in every run.
• Tested on other boxes too: a Threadripper PRO 5975WX with two CMP 170HX (mercant1982) and a Ryzen 9 9950X with an RTX 5090 and two RTX 3090 on mixed links (sanasol: fresh prompts -17 to -34%, agent turns -10 to -29%, decode the same) - see the comments.

What I could and couldn't test

• Here: the box above, with two GPUs and with one. No Windows, no AMD, no tensor-parallel group, no single-socket box - hence opt-in.
• The AMD build: python tools/check_hip_compat.py passes; I could not compile or run HIP here.
• Greedy output across runs: with experts on the CPU lane, decode is not repeatable run to run on this box even with the setting unchanged (ds4: routed experts on the CPU lane make output non-repeatable (one conversation too), which is why --parallel fails --multi-check on 2 GPUs #644), so I compared the prompt pass instead (the first token's top-10 log-probabilities, above).

🤖 Generated with Claude Code

[mercant1982]

I tested 5ea06e1 on a Threadripper PRO 5975WX (32 CPU threads, 512 GiB RAM) with two CMP 170HX 64 GiB GPUs at a configured 150 W each. This uses an existing mixed IQ2_XXS GGUF pack for DeepSeek-V4.1-Flash-1017, TP=1, with CPU-lane experts in RAM. No weight conversion was needed.

Both arms use the same PR binary, fixed read-only expert profile, 32K context, parallel=1, and MAYA_DS4_PROMPT_PICK=length:96. The only feature change is CPU prefill off/on. The on arm reports DS4_PREFILL_CPU activity on both GPUs (including nonzero CPU experts and rows); the off arm reports none. Decode's separate streaming calibration reports zero streaming layers in both arms, which does not prevent this prefill path from running.

For one sequential off/on pass, native prompt time was 2566.8/1927.1 ms for a fresh 1024-token prompt and 5274.2/5040.6 ms for a fresh 4096-token prompt (-24.9% and -4.4%). The repeated 4096-token prompt reused 4095 tokens and took 35.1/39.1 ms. Decode for the fresh prompts was 2652.9/3382.5 ms and 2665.1/2729.5 ms respectively, with 128 generated tokens each. Token hashes differ between off/on, so these are timing observations, not a claim of identical outputs or improved model quality. This is a single-pass applicability check, not a tuned performance result.

Results and exact configuration: results and sanitized configuration.

[PrimusInterParis]

Thanks for testing it on a very different box! I repeated the decode question here (same machine as in the PR description, the PR's code with DS4_FLOW_COPY_DMA off, so only this PR's setting differs): four fresh engine starts interleaved off / on / off / on, the same frozen expert profile, and per run five fresh ~1,000-token prompts plus one ~4,100-token prompt, each answered with 200 greedy tokens (thinking off).

• ~1,000 tokens (10 reads per arm): prompt 10.9 -> 6.9 s, decode right after 20.2 -> 21.3 tok/s, VRAM hit rate 76.6 -> 78.7%
• ~4,100 tokens (2 per arm): prompt 14.2 -> 13.4 s, decode 24.1 -> 24.7 tok/s

So no decode penalty after the CPU share here, and the share only runs while a prompt chunk is read. One thing that may explain your single pass: the first request after an engine start decodes much slower and varies a lot while the expert cache is still cold - 13.7 / 14.0 / 13.7 / 17.0 tok/s for the first request of our four runs, 20-25 tok/s for the later ones. If your 1,024-token prompt was the first request of each arm, that alone could account for 2.65 vs 3.38 s. A warm-up request first, or a few prompts per arm, should show it.

On your box the defaults are probably far off, which would also explain the smaller prompt gain. They were measured here: MAYA_DS4_PREFILL_CPU_EXPERT_MS 0.9 is about one 11 MB expert over a PCIe 3.0 x16 link at ~11 GB/s, and MAYA_DS4_PREFILL_CPU_ROW_MS 0.49 is an Ivy Bridge Xeon without AVX2. As far as I know the CMP 170HX link is throttled to x4 at Gen 1 or Gen 2 (~1-2 GB/s) unless modded, so streaming an expert costs you several times more, while the 5975WX computes a row faster. If you have time to try it: set EXPERT_MS to your expert size divided by your link's rate (the load log prints the link: prompt ring: ... for its X GB/s link), and ROW_MS to cpu_ms / rows from the DS4_PREFILL_CPU line (DS4_PREFILL_STATS=1). I'd expect a clearly larger share for the CPU and a bigger gain at 1K.

[mercant1982]

Thanks for the follow-up. Each arm did have a 256-token / 32-generated-token warm-up before the fresh 1024-token request, although that may still leave the expert cache under-warmed. I agree that this single pass does not establish a decode penalty.

A hardware detail: these cards report PCIe Gen2 x16 (both current and maximum). The same run's Maya load log reports a 6.8 GB/s link on each GPU, with prompt-ring expert slots of 10.1 MB and 11.5 MB. For the fresh 1K prefill, the CPU diagnostics are 69.5 ms / 521 rows and 56.0 ms / 420 rows, about 0.133 ms/row on each GPU; at 4K they are about 0.111 and 0.115 ms/row.

Those measurements support trying different planner estimates. Calibrated off/on timings are still unmeasured here; the published numbers remain the default-estimate applicability check.

[sanasol]

Tested 5ea06e1 on a third, quite different box: one consumer CPU (AVX-512), three GPUs on a layer split with mixed PCIe links. Only this PR's setting differs between the arms.

Box: Ryzen 9 9950X (16C/32T, AVX-512, one socket), 96 GB DDR5-6000 (dual channel), Linux, driver 615.71, CUDA 13.1. GPUs: RTX 5090 32 GB (PCIe 5.0 x8, the engine's ring line: 28.9 GB/s), RTX 3090 24 GB on PCIe 4.0 x4 (OCuLink, 6.7 GB/s), RTX 3090 24 GB on PCIe 4.0 x8 (13.4 GB/s).

Model / engine: DeepSeek-V4.1-Flash-1017 Maya-XS (v2), v1.1.61 (c5842c5) + this PR cherry-picked, --tp 2 (falls back to a layer split, one GPU a group) with DS4_SPLIT=18,9,13 (fewer layers on the x4 card), --parallel 2, --max-context 524288, DSpark on, DS4_RING_MB=400, 16 CPU threads. 9,250 experts in VRAM, 6,578 (47 GB) copied into RAM and pinned.

Harness (as in the PR description): the same binary in both arms, only MAYA_DS4_PREFILL_CPU=0/1 differs; a fresh engine per run, one frozen expert-profile.bin restored before every start, MAYA_DS4_SLOT_KEEP=0, MAYA_DS4_PROMPT_PICK=length:96, DS4_PREFILL_STATS=1 in both; order off / on / off / on, the same requests every run. Per run: two warm-up requests, three fresh prompts (nothing reusable, first token only), a ~24.5K-token start read once, then six appends of 590-3,840 new tokens on it, three fixed prompts for the first token's top-10 log-probabilities, three 300-token greedy answers. Times are the engine's prompt N = R reused + X read in T ms lines.

prompt	without (=0), s	with (=1), s	change
396 tokens, fresh	2.01 / 1.86	1.21 / 1.36	-34%
700 tokens, fresh	2.51 / 2.56	1.71 / 1.66	-34%
1,984 tokens, fresh	3.52 / 3.61	3.02 / 2.87	-17%
24,453-token start, cold	17.03 / 16.86	13.92 / 13.95	-18%
agent turn: 587 new tokens	2.12 / 2.16	1.51 / 1.51	-29%
agent turn: 681 new tokens	2.46 / 2.46	1.86 / 1.86	-25%
agent turn: 850 new tokens	2.56 / 2.61	2.01 / 1.97	-23%
agent turn: 1,344 new tokens	3.06 / 3.06	2.36 / 2.36	-23%
agent turn: 3,780 new tokens	4.38 / 4.41	3.96 / 3.95	-10%
agent turn: 3,838 new tokens	4.36 / 4.48	3.99 / 4.01	-10%

Two runs per arm, each value one run; the two runs of an arm agree within 0.15 s.

• The 24.5K cold start is 18% faster too. In the PR description a 20.6K cold read was unchanged ("chunks of 8,192: no share"). Here the prompt path reads in chunks of 4,096 (DS4_PREFILL ... chunk=4096 on every read of this run, the auto chunk for this split and VRAM), which is exactly MAYA_DS4_PREFILL_CPU_MAX_T, so every chunk of the cold read gets a share (one GPU's line for that read: layers=78 experts=2746 rows=12017 cpu_ms=2657 wait_ms=3078).
• Decode, 300-token answers (tok/s, every answer): without 55.8 / 79.8 / 70.0 / 60.3 / 77.4 / 68.6, with 57.4 / 77.6 / 67.3 / 57.2 / 77.8 / 68.5 - means 68.7 / 67.6, the same within the spread.
• Where the share goes (DS4_PREFILL_STATS, one 3.8K append): the 5090 group (18 layers) took 953-1,091 experts / 2,475-3,576 rows (cpu_ms 657-754), the x8 3090 570-589 experts (cpu_ms 251-264), the x4 3090 only 108-115 experts (cpu_ms 41-49). wait_ms was 0-29 ms except one 360 ms on the 5090 group (the CPU the limit there), lane_busy=0 on the first runs. With the default ROW_MS 0.49 the x4 card, the slowest link, gets the smallest share - the planner weighs the stream by EXPERT_MS alone, not by the GPU's link. A per-GPU expert time (the ring line already measures each link) would probably shift more experts off the x4 card.
• Answers: first-token top-10, KL(without ‖ with) over the common tokens: 1,759 tokens 2.2e-2 to 6.4e-2 (without vs without, across a restart: 1.2e-2); 744 tokens 3.3e-2 to 8.2e-2 (without vs without 8.1e-2); 377 tokens 4.3e-2 to 4.0e-1 (without vs without 1.6e-1). The first token was the same in every run at 744 and 1,759 tokens; at 377 tokens "English" and "The" sit 0.1-0.5 nats apart and swapped between restarts in both arms, not only with the setting. So the drift is the same size as this engine against itself across a restart on this box.

Not tested: MAYA_DS4_PREFILL_CPU_EXPERT_MS / ROW_MS sweeps, DS4_FLOW_COPY_DMA together with it (separately: with DS4_FLOW_COPY_DMA=1 DS4_AI_RUN=1 DS4_YOCO_FLOW=1 on top, agent turns of 0.6K / 1.2K new tokens went 1.6 / 2.2 s -> 1.3 / 2.0 s with the setting, one run each), Windows, tensor-parallel groups.

[PrimusInterParis]

Thanks @sanasol for the third box and the per-GPU breakdown. Both estimates the planner uses are now measured where they are not set (152dc6e, 20e2451):

• MAYA_DS4_PREFILL_CPU_EXPERT_MS unset: each GPU plans with its own expert time, the layer's expert bytes over the link the load already measures (link_h2d, the prompt ring line's figure), in steps of 0.05 ms. The load prints it per GPU (DS4_PREFILL_CPU: a streamed expert X-Y ms at its Z GB/s link). On your 5090 + x4/x8 3090 split that should give the x4 card about four times the 5090's expert time, so the larger CPU share; on Gen2 x16 cards at 6.8 GB/s about 1.5-1.7 ms instead of 0.9.
• MAYA_DS4_PREFILL_CPU_ROW_MS unset: the load times the CPU lane once on 64 of a layer's lane experts for one and two rows, the share's usual shape (DS4_PREFILL_CPU: the CPU lane X ms a row). The 0.49 default came from this Ivy Bridge pair; a 5975WX should come out near the 0.11-0.13 ms a row in @mercant1982's DS4_PREFILL_CPU lines.

Either setting given still applies to every GPU as before. Here (PCIe 3.0 x16: 0.85-1.10 ms an expert, 0.48-0.49 ms a row measured) it plans like the hand-set values: the same prompt times through the staging, agent turns about 2% slower than the hand-tuned 1.2 with DS4_FLOW_COPY_DMA=1. Both links here are the same, so the per-GPU difference itself is untested on this box - a rerun of your arms with both unset would show whether it moves the share off the x4 card.

One GPU, since most setups have one (one RTX 3090, Maya-S, 256K context, --parallel 1, both values measured at the load, two runs per arm, the same frozen profile): fresh prompts of 326 / 602 / 1,638 tokens 8.86 / 10.55 / 13.05 -> 5.11 / 6.51 / 10.05 s (-42 / -38 / -23%), eight agent turns of 506-3,236 new tokens 96.5 -> 66.2 s together (-31%; 500-700 new tokens 10.9 -> 6.8 s). With DS4_FLOW_COPY_DMA=1 (#958) as well: 4.00 / 5.20 / 7.73 s and 56.0 s. Decode the same, the first token the same in every run.

A correction to my earlier comment: I had assumed x4 links for the CMP 170HX; your cards' Gen2 x16 / 6.8 GB/s figures are what the new default works from.

[mercant1982]

Checked 20e245116007 on our dual CMP 170HX/5975WX setup with the existing IQ2 pack. With both overrides unset, both links report 6.8 GB/s, streamed-expert estimates are 1.10-1.55 ms on GPU 0 and 1.55-1.75 ms on GPU 1, and the 64-expert/96-row CPU probe measures 0.07 ms per row. Explicit 0.49/0.9 overrides still suppress automatic cost measurement. Both arms report CPU-prefill work on both GPUs and lane_busy=0.

In one ordered smoke pair, fresh 1024/4096-token prompts took 1915/4996 ms with explicit old costs and 1665/4739 ms with measured costs. Other CPU validation jobs ran during the session, so these are descriptive observations, not an isolated performance comparison. Greedy token hashes differed on three of four cases; quality was not evaluated. These equal-bandwidth links do not validate the mixed-link case.

Driver limits remained 150 W throughout the samples; one instantaneous power reading was 216.47 W. The highest sampled core/HBM temperatures were 54/59 C. No production setting was changed.

Evidence: MAYA.json; MAYA_FIXTURES.json; MAYA_OPERATING.json; maya_worker.py.

Reproduction notes · Evidence bundle and licenses.