[PrimusInterParis]

TL;DR: On a PC with two CPU sockets, Maya keeps its RAM copy of each expert split between the two sockets' memory. Before the graphics card could use such an expert for a prompt, a helper thread first copied both halves into one buffer, and only then did the card fetch it. With DS4_FLOW_COPY_DMA=1 the card now fetches the halves straight from where they lie, as it already could for a copy in one piece. On my PC a short prompt is read 11-16% faster, with exactly the same results. It stays off unless you turn it on.

What

DS4_FLOW_COPY_DMA=1 (opt-in, as before) DMA'd a streamed expert from its pinned RAM copy only when the copy is in one part (NH == 1). Where --mmap-experts copies the CPU lane's experts into RAM split across the CPU sockets (NH parts, "copied into RAM (... 2 NUMA parts)"), each expert still went through the readers' memcpy into the pinned staging and a DMA out of it.

This PR adds the NH > 1 case for a layer alone on its device (l.nm == 1): flow_push marks the item with its copy's slot (Item::cps), and flow_pump copies it with flow_copy_parts, one cudaMemcpyAsync per matrix and part (six for two sockets), each matrix's rows from the part that holds them, side by side in the ring slot - ram_to_blob's layout, the one flow_copy puts whole matrices in, and what copy_streamed already does off the flow. Tensor-parallel members keep the staging (they read a member's rows and its columns of the down). Without the setting nothing changes. Docs: the setting's bullet and row in docs/SETTINGS.md.

Why

Through the staging every byte crosses the RAM three times (the reader's read, its write into staging, the DMA's read). On DDR3 that is the prompt's limit: the flow streamed at ~7.8 GB/s effective on a PCIe 3.0 x16 link that does 11.0-11.6 GB/s from pinned memory (a plain cudaMemcpyAsync test, 1-256 MB pieces, either socket's memory). With the DMA in place the stream runs at the link's rate: with DS4_PROF=1 on v1.1.56 and the CPU share off, a fresh 326-token prompt's experts took 7.4 s of GPU time through the staging and 6.1 s with the DMA (attention and routing ~0.35 s either way) - 1.0 ms an 11.2 MB expert, the link's ~11 GB/s.

Measured

Two Xeon E5-2697 v2 (Ivy Bridge, 12 cores each: AVX and F16C, no AVX2 or FMA), 16x 32 GB DDR3-1600, 2x RTX 3090 24 GB on PCIe 3.0 x16 (both on socket 0); Linux, CUDA 13.0. Maya v1.1.59 (027992c) + this PR (rebased onto main since; only docs changed there), DeepSeek-V4.1-Flash-1017 Maya-S, --tp 2 (falls back to a layer split: one GPU a layer group), --parallel 2, --max-context 1048576, --dspark none, 20 CPU threads; 3,144 experts in VRAM, 12,212 (134.6 GB) copied into RAM in 2 NUMA parts and pinned.

Harness: a fresh engine per run, one frozen copy of expert-profile.bin restored before every start (the profile the server saves at each stop moves the placement otherwise), the same requests every run, MAYA_DS4_SLOT_KEEP=0, MAYA_DS4_PROMPT_PICK=length:96 in every run (so a run's path does not depend on the timing pick). Per run: three fresh prompts (326 / 602 / 1,638 tokens, nothing reused, first token only), then a 20.6K-token start read once and eight reads on top of it (an agent's turn: 500-3,200 new tokens), then the decode bench. Runs in this order: neither setting, DS4_FLOW_COPY_DMA=1 only, MAYA_DS4_PREFILL_CPU=1 only (#959), both, neither again; "without" below is the mean of the two runs with neither. The first run after the previous engine stopped differs from all later ones in its last bits (with every setting off too); every later run with the same arithmetic is bit for bit the same.

prompt	without (=0)	with (=1)	change
326 tokens, fresh	7.55 s	6.70 s	-11%
602 tokens, fresh	8.95 s	7.52 s	-16%
1,638 tokens, fresh	11.53 s	9.82 s	-15%
agent's turn: 506-708 new tokens on the 20.6K start (4 reads)	8.90-9.66 s	7.54-8.29 s	-14 to -15%
1,141 / 1,152 new tokens	10.20 / 10.43 s	8.78 / 9.12 s	-14 / -13%
3,123 / 3,236 new tokens	12.67 / 12.91 s	11.10 / 11.34 s	-12%
20.6K tokens, cold (chunks of 8,192)	25.49 s	22.87 s	-10%

With MAYA_DS4_PREFILL_CPU=1 on in both arms (#959, MAYA_DS4_PREFILL_CPU_EXPERT_MS 0.9 without, 1.2 with): fresh 326 / 602 / 1,638 tokens 4.29 / 5.15 / 8.12 -> 3.18 / 3.92 / 6.59 s (-26 / -24 / -19%), an agent's turn of 506-708 new tokens 5.36-6.03 -> 4.08-4.94 s, 3,123 / 3,236 new tokens 9.91 / 10.05 -> 8.74 / 9.03 s.

Decode (three answers of 300 tokens, greedy, thinking off, two sets of questions A / B): 24.5 / 25.8 and 26.2 / 27.2 tok/s without (two runs), 25.5 / 27.7 with - the same within the runs' spread.

Answers: the prompt pass is bit for bit the same - the first token's top-10 log-probabilities of the three fresh prompts are identical with and without it, and between repeated runs of each. Only the copies change, not the arithmetic.

Related: #910 documents the one-part case (one GPU, AMD); this is the multi-socket one. #747 tried the same direct copies on a DDR4 box and was closed for a gain within the noise; one GPU and a full greedy-output check for this one are in the comment below.

What I could and couldn't test

• One setup only: the box above. No Windows, no AMD, no tensor-parallel group, no single-socket box - hence opt-in.
• The AMD build: python tools/check_hip_compat.py passes; I could not compile or run HIP here.
• Greedy output across runs: with experts on the CPU lane, decode is not repeatable run to run on this box even with the setting unchanged (ds4: routed experts on the CPU lane make output non-repeatable (one conversation too), which is why --parallel fails --multi-check on 2 GPUs #644), so I compared the prompt pass instead (the first token's top-10 log-probabilities, above).

🤖 Generated with Claude Code

[PrimusInterParis]

#747 tried the same direct copies and was closed for a gain within the run-to-run spread and no greedy-output check, so here are both for this one, measured since on the box from the description:

One GPU (one RTX 3090 on PCIe 3.0 x16, Maya-S, --max-context 262144, --parallel 1, 155.8 GB of experts copied into RAM in 2 NUMA parts; a fresh engine per run, the same frozen expert profile, two runs per arm, order base / DMA / ... / DMA / base):

• fresh prompts of 326 / 602 / 1,638 tokens: 8.86 / 10.55 / 13.05 -> 7.85 / 9.07 / 11.23 s (-11 / -14 / -14%)
• eight agent turns of 506-3,236 new tokens on a 20.6K start: 96.5 -> 84.3 s together (-13%)
• the two runs of an arm within 0.25 s of each other; decode the same (17-19 tok/s either way)
• the first token's top-10 log-probabilities identical with and without, and between the two runs of each arm

Greedy output: decode on this box is not reproducible across restarts even with identical settings (the expert cache's moves depend on timing, and a CPU-lane expert's sum differs in its last bits from the GPU's). With --no-expert-cache the placement is fixed, and then six greedy answers of 300 tokens (three German prose, three code; two GPUs, thinking off) are identical token for token and log-prob for log-prob with the setting off / on / off. Only the copies change, not the arithmetic.

Why it shows here: through the staging every streamed byte crosses the RAM three times. On DDR3-1600 that held the stream at ~7.8 GB/s on a link that does 11-11.6 GB/s from pinned memory. #747's box had eight-channel DDR4 per socket, where the staging is probably not the limit, which would explain the small gain there.