Repository navigation
DeepSeek: the automatic layer split balances streamed bytes, not time, so short prompts wait on the slow-link GPUs #733
Description
DeepSeek's automatic layer split (the "contiguous split that streams the least into its busiest member", ds4_model.cu) balances the bytes each GPU streams, not the time. With mixed links that leaves the slow cards as the bottleneck, and short prompts pay it most: a prompt of a few thousand tokens is one chunk, its stages run one after another, so the read takes about the sum of each card's streaming time.
Measured on 4x RTX 3090 (PCIe 3.0: CUDA0 and CUDA2 x8 at 6.7 GB/s, CUDA1 and CUDA3 x16 at 13.2 GB/s), Maya-S, v1.1.42, --parallel 2. DS4_PROF on a 1,015-token read on top of a reused conversation (8.2 s):
| CUDA0 x8 | CUDA1 x16 | CUDA2 x8 | CUDA3 x16 | |
|---|---|---|---|---|
| streamed from RAM | 18.9 GB | 15.8 GB | 17.9 GB | 16.5 GB |
| compute waited for reads | 1.7 s | 0.6 s | 1.5 s | 0.8 s |
Attention and the rest of the compute are under 1 s. The auto split is 8, 11, 11, 10. Moving layers from the x8 cards to the x16 ones with DS4_SPLIT (fresh reads of N tokens on a reused 20K-token conversation, 2 each, no overlapping requests):
| new tokens | auto 8,11,11,10 | DS4_SPLIT=7,12,9,12 |
6,13,9,12 |
|---|---|---|---|
| ~256 | 5.0-5.6 s | 4.3-4.5 s | 3.8-4.4 s |
| ~1K | 8.9 s | 7.5-7.6 s | 7.5-7.6 s |
| ~2K | 10.4-10.6 s | 9.2 s | 9.1 s |
| ~4K | 12.2-12.9 s | 11.6-12.0 s | 10.6-11.5 s |
| 101K, tok/s | 1,223-1,232 | 1,252-1,258 | 1,247-1,250 |
| decode, tok/s | 38.1-40.0 | 38.4-39.0 | 37.6-37.8 |
We run 7,12,9,12 now: 256-2K-token reads, most of an agent's turns, are 13-17% faster.
Possible fix: weight each member's cost() by its measured host link (the ring sizing already measures it), so a split minimizes streaming time rather than bytes.
A second idea, for short chunks: GLM computes the least-routed RAM-tier experts on the CPU while the GPU streams the rest (MAYA_GLM_PREFILL_CPU); DeepSeek streams them all. Here the CPU lane reads RAM at ~27 GB/s against 6.7 GB/s on the x8 links, so the same split for DeepSeek's prompt path could cut most of the x8 cards' waits. Happy to test either on this machine.
Activity
mw00 commentedon Nov 9, 2026
Thanks for the measurements. Balancing bytes instead of time on mixed links makes sense as an explanation for slow short prompts. Not decided yet, but we will look at weighing each GPU's link speed in the split. Please tell us if v1.1.57 changes the picture.
sanasol commentedon Nov 12, 2026
A second machine for this, on v1.1.61 (so after v1.1.57): the picture is the same. The split still balances bytes, and on mixed links the slowest one sets the pace of short reads.
Box: Ryzen 9 9950X (16C/32T), 96 GB DDR5-6000, Linux, CUDA 13.1. Three different links, as the engine measures them (
prompt ring ... for its N GB/s link):Model / config: DeepSeek-V4.1-Flash-1017 Maya-XS v2,
--tp 2(falls back to a layer split),--parallel 2,--max-context 524288, DSpark on. 9.2k experts in VRAM, 6.6k (47 GB) in RAM. The auto split puts the 3090 x4 second, as GPU 1.1. Without the skip (v1.1.61 defaults; warm runs; cold 24K read / a 2.4K-token append on a 24K conversation / decode):
DS4_SPLIT=16,11,13DS4_SPLIT=18,9,13+DS4_RING_MB=400Fewer layers on the x4 card: short appends -18%, decode unchanged. Same direction and size as your 4x3090 table.
2. With
DS4_FLOW_COPY_DMA=1 DS4_AI_RUN=1 DS4_YOCO_FLOW=1and #959'sMAYA_DS4_PREFILL_CPU=1, the best split moves. With the skip, prompt reads put most of the work in layers 0-19. Medians of two rounds, which agree within 1%; agent turns are appends on a 24K start:So two things pull the split in opposite directions:
We now run 16,8,16 as a compromise: cold reads +8-13% over 18,9,13, appends unchanged. Decode is the same for every split, within ±1.5 t/s.
For the planner, a cost per GPU of roughly
streamed bytes / measured link speed(the ring sizing already has the link), plus the skip's shift of work into the first 20 layers, would land near what we found by hand. The same per-GPU link term would also help #959. There the share each GPU gives the CPU is weighed by oneEXPERT_MS, so the x4 card, which is the slowest link, got the smallest share (~110 experts per prompt vs ~1,000 on the 5090; numbers in my comment there).One more data point: dropping the x4 card entirely (two GPUs, 5090 + 3090 x8) reads a cold 24K prompt 13% faster (1,640-1,680 vs 1,450-1,490 t/s on the defaults). But it decodes 48 instead of 65 t/s, because 2.8k more experts go to RAM. The third card is worth keeping; it just should not get prompt-read work in proportion to its bytes.
mgabor3141 commentedon Nov 13, 2026
Same here on v1.1.61: the split's cost function is unchanged since v1.1.42, and we still pin
DS4_SPLIT=7,12,9,12(not re-measured against auto). @sanasol's numbers above match ours.