DeepSeek V4 Flash on 8× RTX 5090 — 50 sessions × 50k KV resident

This repository contains the configuration, kernel patches, and benchmarks for running 0xSero/DeepSeek-V4-Flash-0731-REAP (REAP-pruned, 160/256 experts) on eight consumer RTX 5090 32GB cards (SM120) with SGLang — tuned for many concurrent long-context sessions, not single-stream speed.

Results from the pinned stack (SGLang 0.5.16, flashinfer 0.6.14 + topk=192 backport, no speculative decoding):

Metric	Result
50 sessions × 50k context, all KV resident	1,578 tok/s aggregate (29.4 tok/s median per stream, TTFT p50 3.0 s)
Aggregate ceiling, short contexts	1,718 tok/s @50 · 1,971 @56 · 2,058 @64 sessions
40 × 50k resident	1,606 tok/s aggregate (36.8/stream)
Cold prefill	~6,500–8,600 tok/s (50k prompt ≈ 6–8 s)
KV pool	2,499,840 tokens (2.5M cap)
Weights in VRAM	13.95 GiB/GPU (REAP K160) vs 20.45 GiB/GPU for the stock fp8+mxfp4 checkpoint
Speculative decoding at 40+ sessions	a net loss — full DSpark block curve below

The same serving goal on the stock checkpoint topped out at ~1.0–1.5M resident tokens (~32 sessions × 50k with CUDA graphs) before this work; the pruned checkpoint plus the configuration findings in this repo are what make 50×50k fit.

---

Why 8× 5090

An 8× RTX 5090 (32GB) box rents for ~$2.7–3.8/hr on demand (Vast.ai, August 2026) — a fraction of equivalent datacenter GPUs — but the economics only work if you respect the wall:

• 32GB per card is the whole game. DSV4's MLA KV latent is replicated on every TP rank (num_key_value_heads=1), so TP=8 does not multiply KV capacity. VRAM per GPU is the only currency, and every spare GB of weights maps 1:1 to KV tokens.
• No NVLink: all TP traffic crosses PCIe. Measured at high batch this is 3–6% of step time — not the bottleneck. Don't pay a premium for PCIe 5.
• The catch: SM120 (consumer Blackwell) needed real kernel work to serve this model at all (see the patch), and the memory wall forced every decision below.

Prior art, and what this repo adds

• The REAP checkpoint (0xSero) prunes 37.5% of routed experts (256→160, top-6 routing preserved) with router-identity-aligned transferred rankings. 107.8GB on disk vs ~163GB — that is the capacity unlock.
• flashinfer#4309 adds the SM120 topk=192 sparse-MLA kernels this model's DSA needs (128 SWA + DSpark block 5, padded to a 64-tile). Open at time of writing; this repo carries it as a backport onto 0.6.14.
• Our own prior worklog on the stock checkpoint (same hardware) established the KV ceiling map, the SWA pool as a first-class constraint, and the extend-indexer OOM death modes.

What this repo adds:

1. A validated 50 × 50k resident recipe on 8× 32GB consumer cards — previously impossible at any quant; the stock checkpoint physically cannot do it (~34GB/GPU needed).
2. The no-spec finding: with CUDA graphs, disabling DSpark beats every spec block size at 40+ concurrent sessions (1,778 vs 1,150/1,020/761 agg tok/s at 40, empty KV), and frees ~1.4GB of verify-graph memory — which is exactly what lets 50×50k fit. Spec only wins at low batch (41 vs 8.7 tok/s single-stream).
3. Three production traps, root-caused (see Production notes): the radix-retention SWA floor, the first-run kernel-warmup tax, and Triton's raw-cudaMalloc decode OOM.
4. A Docker recipe that patches, compiles, and validates the topk=192 kernels with the PR author's own 47 tests on first boot.
5. Honest dead ends: torch.compile (graph capture invalidated), HiSparse (requires --disable-radix-cache), NVFP4 checkpoints (175.6GB — larger than stock), 42–45×50k spec-on configs (prefill-storm OOM).

Repository layout

.
├── Dockerfile                  # lmsysorg/sglang base + topk=192 source patch
├── docker-compose.yml          # the production stack (gpus: all, ipc: host)
├── SHA256SUMS                  # pins for every runtime artifact
├── scripts/
│   ├── entrypoint.sh           # patch -> kernel validation -> model download -> serve
│   ├── start_server.sh         # THE validated configuration (see flag table)
│   ├── validate_topk192.sh     # compiles kernels, runs the PR author's 47 tests
│   ├── model_present.py        # HF snapshot check
│   ├── retention_probe.py      # swa/retention sanity gate (<1.5s resend required)
│   └── decode_bench.py         # prewarmed decode-only TPS bench (always run twice)
├── patches/
│   ├── flashinfer-topk192.patch    # unified diff vs upstream 0.6.14 (PR #4309 content)
│   ├── patch_flashinfer.py         # idempotent applier for the installed package layout
│   └── README.md                   # provenance and regeneration instructions
├── tests/
│   └── test_sparse_mla_sm120.py    # PR #4309 author's kernel tests (SM120, topk=192)
└── docs/
    └── findings.md             # full work log: KV ceiling map, death modes, tuning tables

Runtime configuration

The stack serves the REAP K160 checkpoint with no speculative decoding:

Flag	Value	Why
--tp 8		8 cards; KV replicated per rank (see above)
--moe-runner-backend flashinfer_mxfp4		native MXFP4 experts — the 13.95 GiB/GPU footprint
(no --speculative-algorithm)		net loss at 40+ concurrent; also frees verify-graph VRAM
--mem-fraction-static 0.94		pool 2.5M tokens + ~2.7GB transient headroom
--max-total-tokens 2500000		exactly 50 × 50k; profiler would oversubscribe past it
--swa-full-tokens-ratio 0.04		retention floor — at ≤0.03 RadixCache silently stops retaining; every request re-prefills
--chunked-prefill-size 2048 / --max-prefill-tokens 2048		one long extend in flight; tames the ~256MB/50k indexer workspace
--cuda-graph-max-bs-decode 56		covers 50-session decode batches; no-spec capture is small
--max-running-requests 64		headroom above the 50-session operating point

Env: SGLANG_DSV4_COMPRESS_STATE_DTYPE=bf16, SGLANG_SM120_FLASHMLA_BACKEND=flashinfer, PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True, plus the four official SWA eviction knobs (see scripts/start_server.sh).

Deploying it

1. Host prerequisites

8× RTX 5090 32GB visible to Docker (nvidia-container-toolkit), ≥300GB free disk (model ~108GB + image ~35GB), recent Docker Compose. PCIe topology is not critical.

2. Build and start

export HF_TOKEN=hf_...          # needed once, for the 108GB model download
docker compose build
docker compose up -d
docker compose logs -f

First boot takes ~20–30 minutes: topk=192 kernel compile + 47-test validation (~3 min), model download (~10–20 min at ≥1Gbps), weight load + graph capture (~5 min). A healthy start must show:

DSV4 memory calculation: bytes_per_full_token=5517.73, available_bytes=13.84 GB
DSV4 pool sizes: full=2499840, swa=99840, c4=624960, c128=19530, c4_state=6240
Capture target decode CUDA graph end. ... avail mem=~2.7 GB
The server is fired up and ready to roll!

If avail mem after capture is under ~1GB, the server may start but die on the first long-context extend — do not raise --mem-fraction-static or the pool cap.

3. Smoke-test

KEY=$(docker exec dsv4-reap cat /root/.cache/sglang_api_key)
curl -fsS http://localhost:30000/v1/models -H "Authorization: Bearer $KEY"

# retention gate — MUST print RESEND < 1.5s (~6s means misconfigured; see traps)
docker exec dsv4-reap python3 /root/scripts/retention_probe.py

4. Benchmark

# ALWAYS run twice — the first run after any launch pays a one-time ~30s kernel
# warmup and reads garbage (streams_ok ~10, TTFT ~30s). The second is the truth.
docker exec dsv4-reap python3 /root/scripts/decode_bench.py \
  --url http://127.0.0.1:30000 --model 0xSero/DeepSeek-V4-Flash-0731-REAP \
  --api-key "$KEY" --levels 50 --context-tokens 50000 \
  --output-tokens 512 --duration 60 --disjoint --out /root/bench.json

The patch

patches/flashinfer-topk192.patch backports PR #4309 onto flashinfer 0.6.14: five DSV4_DISPATCH(H, 192) instantiations in the SM120 decode kernel, a topk==192 BF16 branch in the prefill dispatcher, and five (H,192) tuples in the Python dispatch table. Stock 0.6.14 instantiates only topk ∈ {128, 512, 1024, 2048} for SM120 DSV4 — without this, SGLang crashes the moment DSA's topk=192 path is exercised.

Two installation subtleties, both handled by patch_flashinfer.py and the entrypoint:

• The installed package keeps sources under flashinfer/data/csrc/... (not csrc/).
• The flashinfer-jit-cache package ships a prebuilt sparse_mla_sm120.so that shadows locally built kernels — it must be moved aside (and the rebuilt .so copied in), or your patch compiles but never loads. Traceback paths containing /workspace/... are the tell.

Validation: the PR author's own pytest slice, 47/47 passed on 8× 5090.

Performance

All numbers are decode-only, KV prewarmed, prefill excluded (decode_bench methodology), disjoint 50k contexts unless marked empty-KV, REAP K160, 8× 5090.

Headline config (no-spec, graphs bs56, pool 2.5M):

Sessions × context	Aggregate tok/s	Median per-stream	TTFT p50
50 × 50k	1,578	29.4	3.0 s
40 × 50k	1,606	36.8	2.5 s
50 × empty	1,718	33.3	0.8 s
56 × empty	1,971	33.8	0.8 s
64 × empty	2,058	31.4	0.9 s

DSpark block sweep (40 sessions, empty KV, graphs bs56):

Config	Aggregate tok/s
no spec	1,778
block 1	1,144–1,159
block 2	1,140–1,157
block 3	1,020
block 5	761

At 40+ sessions the verify batch costs more than acceptance buys; at bs≈1 the same model gains ~4–5× from block 5. Pick your operating point and measure both.

Eager vs graphs (49×50k resident, block 3): 388 agg tok/s eager vs 872–880 graphed at 40 — graphs are the 2.3–4× lever at this scale; eager is only the capacity fallback.

Measured dead ends: torch.compile (cudaErrorStreamCaptureInvalidated at capture); HiSparse (requires --disable-radix-cache — incompatible with warm-session serving); NVFP4 community checkpoints (175.6GB — larger than the stock fp8+mxfp4 mix); 42–45 × 50k with spec on (prefill-storm OOM at 388–556 MiB free).

Production notes

• Trap 1 — radix retention has an SWA floor. At --swa-full-tokens-ratio ≤ 0.03, RadixCache silently stops retaining: a resent 40k prompt takes a full ~6s re-prefill instead of a 0.4s hit. 0.04 retains. Bisected config-by-config; chunk size and the SWA eviction knobs are innocent. Always run retention_probe.py after changing pool flags.
• Trap 2 — the first bench after every launch is garbage. One-time ~30s kernel warmup on the first real decode batch: streams_ok ~10, TTFT ~24–32s, TPS ≈ 0.3. Rerun and it reads 50/50 at full speed. Warm the server before admitting traffic.
• Trap 3 — decode death is Triton's raw cudaMalloc. Triton kernel workspace is allocated outside PyTorch's reserved pool, so expandable_segments doesn't help and only nvidia-smi-visible free memory counts for it (~400MB at 50-way decode). Prefill extends (~256MB per 50k) come from torch's pool — a config can survive the prefill storm and still die on first decode. Keep ≥500MB truly free.
• The REAP quality gate is on you. The K160 rankings were transferred from a preview revision observation run (router-row alignment cosine ≥0.99), not a fresh 0731 observation. Sanity output is coherent; run your own coding/agentic evals before production traffic. SGLang routes 160 experts natively; vLLM needs the repo's runtime/patch_vllm_router_k160.py.
• Capacity math (measured): 5,518 bytes/token at swa 0.04; weights 13.95 GiB/GPU; c128_state 0.65GB at maxrun 64; no-spec graph capture ~1.35GB. 2.5M tokens is the 32GB wall; 64 × 50k resident needs the #29927-generation memory work or bigger VRAM.
• Watch upstream: sglang#29927 (DeepGEMM paged-MQA indexer + SM120 GEMM stack, TPOT 25.2→13.3ms bs1 on PRO 6000) and the merge of flashinfer#4309 (retires the local patch). Both change the numbers on this box.

License and provenance

Stack, docs, and scripts are Apache-2.0 (see LICENSE). patches/flashinfer-topk192.patch carries the PR #4309 change (BSD-3-Clause flashinfer project, NVIDIA) — regenerate per patches/README.md. The model is MIT-licensed (DeepSeek); the REAP checkpoint is by 0xSero (MIT). Benchmark figures are our own measurements on the pinned stack (rented 8× 5090, 2026-08-02) — treat them as gates for this configuration, not universal model numbers. Repo structure inspired by ryanzhou/deepseek-v4-flash-mi300x.

References

All links verified 2026-08-02.

• 0xSero/DeepSeek-V4-Flash-0731-REAP — the REAP K160 checkpoint this stack serves (160/256 experts, DSpark drafter preserved)
• deepseek-ai/DeepSeek-V4-Flash-0731 — official model card (284B total, hybrid CSA/HCA attention, MIT)
• flashinfer PR #4309 — SM120 DSV4 topk=192 decode+prefill kernels (carried here as a backport)
• flashinfer issue #3828 — the SM120 topk instantiation gap
• sglang PR #29927 — DeepGEMM paged-MQA indexer + SM120 GEMM/scheduler stack (next big TPS lever)
• sglang PR #23741 — compressed-attention mixed prefill/decode scheduler fix
• SGLang — the serving runtime (0.5.16 pinned)
• ryanzhou/deepseek-v4-flash-mi300x — the single-MI300X vLLM production recipe (format inspiration; different stack)