Skip to content

Commit 0cab21d

Browse files
nilignilig
committed
findings(p2p): add SIG report - best results across four models
Presentation-ready summary: delta methodology per model/accelerator, best results at a glance, and per-run detail (setup + results + delta%) for every run that showed value - gpt-oss docQA/multi-turn/pool/P/D-stack, Llama pool and P/D, Qwen3-30B agentic, GLM-5.2 precise pair - plus the measured boundaries where P2P does not help. Signed-off-by: nilig <nili.ifergan@gmail.com>
1 parent e9a819c commit 0cab21d

1 file changed

Lines changed: 299 additions & 0 deletions

File tree

‎test/p2p-findings/SIG-REPORT.md‎

Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,299 @@
1+
# P2P KV-Cache Sharing — Measured Results (SIG)
2+
3+
Peer-to-peer KV-cache pull in llm-d: when a peer pod holds more cached prefix
4+
tokens than the scheduled pod, the router attaches a KV-source header and the
5+
pod pulls the prefix from that peer (CPU-to-CPU over NIXL/UCX) instead of
6+
recomputing it. Four models (8B dense to 753B MoE), aggregated and P/D
7+
topologies, all NVIDIA H200 (CoreWeave).
8+
9+
Full data, raw logs, and exact configs: `test/p2p-findings/RESULTS.md`
10+
(series 1), `RESULTS-2.md` (series 2: P/D + guide add-on runs), GLM arms
11+
under `configs/glm-5.2-p2p/`, series-2 arms under `configs/series-2/`.
12+
13+
---
14+
15+
## 1. How the delta is calculated
16+
17+
**Protocol.** Every comparison is a paired A/B on identical pods, image, and
18+
workload; the only change between arms is the EPP `--config-file` (the routing
19+
plugin config). Before every measured run: mechanism-engaged gates (prefix
20+
index populated, source header firing, `vllm:external_prefix_cache_hits_total`
21+
rising), full pull-mesh warm-up, `PYTHONHASHSEED` pinned fleet-wide.
22+
23+
**Formula.** For each metric:
24+
25+
```
26+
delta% = (P2P_arm - baseline_arm) / baseline_arm x 100
27+
```
28+
29+
Latency metrics (TTFT, request latency): negative delta = improvement.
30+
Throughput metrics (req/s, tok/s): positive delta = improvement.
31+
Pull activity is verified per run from engine counters
32+
(`external_prefix_cache_hits_total`, `kv_offload_load_bytes_total`), so a
33+
quoted delta is always backed by evidence the mechanism actually ran.
34+
35+
**Pairing per model and accelerator:**
36+
37+
| Model | Accelerator / rig | Baseline arm | P2P arm | `minCachedTokenDelta` | Harness | Repeats |
38+
|---|---|---|---|---|---|---|
39+
| gpt-oss-120b (MXFP4) | 16x H200, TP=1 (14x in multi-turn) | Precise guide blend (prefix 3 / queue 2 / kv-util 2, max-score) — or Load-no-P2P for pool scenarios | Load-aware placement + `p2p-source-producer` | 2,048 (crossover-derived) | inference-perf via llm-d-benchmark | Doc-Q&A: 2 runs, arm order alternated; others single |
40+
| Llama-3.1-8B-Instruct | 4x H200, TP=1 (P/D variant: 4 prefill + 1 decode) | Load, no P2P (identical placement) | Load + P2P | 2,048 (crossover-derived) | inference-perf via llm-d-benchmark | single |
41+
| gpt-oss-120b, P/D | 16x H200: 8 prefill + 8 decode, TP=1 | pd-disaggregation guide verbatim (plain `NixlConnector`) | guide + full P2P stack (`MultiConnector` Nixl + Offloading p2p tier, sidecar `--enable-p2p-pull`, precise producer + `p2p-source-producer`) | 2,048 | llm-d-benchmark docQA profile, C=192 (mechanism-gated) | paired A/B, clean re-roll per arm |
42+
| Qwen3-30B-A3B-Thinking | 6x H200: 2 prefill + 4 decode, TP=1 | agentic guide on plain NIXL | guide + P2P stack | 1,024 (rig-calibrated: 8K pull 74 ms vs 1.21 s recompute, crossover ~760) | llm-d-benchmark agentic synthetic, C=16 | P2P arm sampled twice |
43+
| GLM-5.2-FP8 (753B MoE) | 32x H200: 1P + 1D, wide-EP DEP16 per role | armB: precise affinity, no pull | armC-16k: same precise affinity + `p2p-source-producer` | 16,384 (GLM KV is ~2x heavier per token; 2,048 fires pulls that cost more than they save) | aiperf agentx (Weka agentic traces), c128 | single |
44+
45+
The three delta styles to be aware of:
46+
47+
* **Pool / P/D scenarios (gpt-oss A, Llama):** baseline and P2P arm use the
48+
*same load-aware placement*; the delta isolates the pull itself (recompute
49+
vs pull on every cross-pod miss).
50+
* **Conversational scenarios (gpt-oss C, D) :** baseline is the tuned precise
51+
guide default; the delta measures the *shipping alternative*, so it bundles
52+
placement + pull. GLM's precise pair is again a pure pull isolation.
53+
* **P/D stack runs (gpt-oss C=192, Qwen3-30B agentic):** baseline is the
54+
shipping guide on plain NIXL; the delta measures adding the whole P2P stack
55+
(CPU offload tier + pull). Attribution between tier and pull is stated per
56+
run from the engine counters.
57+
58+
---
59+
60+
## 2. Best results at a glance
61+
62+
| Model | Accelerator | Scenario | Metric | Baseline | + P2P | Delta |
63+
|---|---|---|---|---|---|---|
64+
| gpt-oss-120b | 16x H200 | Document Q&A (128 x ~50K ctx) | TTFT p99 | 80.5 s | 20.9 s | **-74%** |
65+
| gpt-oss-120b | 16x H200 | Document Q&A | TTFT p95 | 41.0 s | 13.0 s | **-68%** |
66+
| gpt-oss-120b | 16x H200 | Document Q&A | throughput | 5.98 turns/s | 7.02 turns/s | **+17%** |
67+
| gpt-oss-120b | 14x H200 | Multi-turn chat (8 turns to 48K) | TTFT p50 | 2.53 s | 1.42 s | **-44%** |
68+
| gpt-oss-120b | 14x H200 | Multi-turn chat | TTFT p99 | 25.4 s | 9.2 s | **-64%** |
69+
| gpt-oss-120b | 16x H200 | Uniform pool (6M-token set) | sustained rate | 9.4 req/s | 16.7 req/s | **+78%** |
70+
| gpt-oss-120b | 16x H200 | Uniform pool @ 12 req/s | req latency p50 | 24.9 s | 2.3 s | **-91%** |
71+
| gpt-oss-120b | 16x H200 | Pull vs recompute, 49K prefix | prefill latency | 1,695 ms | 550 ms | **-68%** |
72+
| Llama-3.1-8B | 4x H200 | Prefix pool 64 x 16K @ 12 req/s | req latency p50 | 12.2 s | 2.1 s | **-83%** |
73+
| Llama-3.1-8B | 4x H200 | Prefix pool, saturation ceiling | achieved rate | 10.3 req/s | 12.6 req/s | **+22%** |
74+
| Llama-3.1-8B | 4x H200 | P/D prefill placement | achieved rate | 11.3 req/s | 14.7 req/s | **+30%** |
75+
| Llama-3.1-8B | 4x H200 | Pull vs recompute, 16K prefix | prefill latency | 503 ms | 155 ms | **-69%** |
76+
| gpt-oss-120b | 16x H200 (8P+8D) | P/D docQA C=192, guide vs guide+P2P stack | TTFT p50 | 11.94 s | 1.16 s | **-90% (10x)** |
77+
| gpt-oss-120b | 16x H200 (8P+8D) | P/D docQA C=192 | TTFT p99 | 106.1 s | 80.0 s | **-25%** |
78+
| gpt-oss-120b | 16x H200 (8P+8D) | P/D docQA C=192 | throughput | 5.68 turns/s | 7.96 turns/s | **+40%** |
79+
| Qwen3-30B-A3B | 6x H200 (2P+4D) | Agentic multi-turn (50K contexts, tool gaps) | TTFT p50 | 5.22 s | 1.09 s | **-79% (4.8x)** |
80+
| Qwen3-30B-A3B | 6x H200 (2P+4D) | Agentic multi-turn | TTFT p95 | 18.94 s | 11.77-14.79 s | **-22..-38%** |
81+
| Qwen3-30B-A3B | 6x H200 (2P+4D) | Agentic multi-turn | throughput (1/duration) | 304 s | 229 s | **+33%** |
82+
| Qwen3-30B-A3B | 6x H200 (2P+4D) | Pull vs recompute, 8K prefix | prefill latency | 1,210 ms | 74 ms | **-94% (16x)** |
83+
| GLM-5.2-FP8 | 32x H200 | Agentic c128, precise routing | TTFT p50 | 3,802 ms | 3,177 ms | **-16%** |
84+
| GLM-5.2-FP8 | 32x H200 | Agentic c128, precise routing | TTFT p90 | 11,755 ms | 9,970 ms | **-15%** |
85+
| GLM-5.2-FP8 | 32x H200 | Pull vs recompute, 24K prefix | TTFT (single req) | 4.63 s | 3.91 s | **-16%** |
86+
87+
---
88+
89+
## 3. Per-run detail
90+
91+
Every subsection: exact setup, results with percentage delta, and the win in
92+
one or two sentences.
93+
94+
### 3.1 gpt-oss-120b — Document Q&A (the headline)
95+
96+
| Setup | |
97+
|---|---|
98+
| Model / pods | `openai/gpt-oss-120b` (MXFP4), 16x TP=1, H200 |
99+
| KV / tiers | ~41.5 KB/token; GPU KV ~1.38M tokens/pod; CPU tier 64 GiB/pod |
100+
| vLLM | nightly + `generic_p2p` OffloadingConnector branch; block size 64 |
101+
| Baseline EPP | precise guide blend: `prefix-cache-scorer` 3 / `queue-scorer` 2 / `kv-utilization` 2, max-score picker |
102+
| P2P EPP | load-aware placement + `p2p-source-producer` (`minCachedTokenDelta: 2048`) |
103+
| Workload | 192 docs x 48K-token private prefix, 6 short questions each, 128 concurrent; 2 full runs, arm order alternated |
104+
105+
| Metric | Precise guide | Load + P2P | Delta |
106+
|---|---|---|---|
107+
| TTFT p50 (run 1 / 2) | 4.1 / 4.2 s | 4.5 / 3.9 s | +10% / -7% |
108+
| TTFT p95 (run 1 / 2) | 41.0 / 17.3 s | 13.0 / 12.5 s | **-68% / -28%** |
109+
| TTFT p99 (run 1 / 2) | 80.5 / 37.2 s | 20.9 / 26.7 s | **-74% / -28%** |
110+
| Throughput (run 1 / 2) | 5.98 / 7.66 turns/s | 7.02 / 7.76 turns/s | **+17%** / +1% |
111+
| Run-to-run throughput spread | 28% | 10% | 2.8x more stable |
112+
113+
**Win:** on prefill-dominated conversational load, displaced questions pull
114+
their 48K context (~0.6 s) instead of recomputing or queueing behind a
115+
concentrated owner (~2 s+): tails drop 2-4x, and the P2P arm is far less
116+
sensitive to inherited cache state (30-32M tokens pulled per run).
117+
118+
### 3.2 gpt-oss-120b — Multi-turn chat
119+
120+
| Setup | |
121+
|---|---|
122+
| Rig | 14x TP=1, `--gpu-memory-utilization=0.60` (sessions exceed fleet GPU cache and evict); CPU tier 88 GiB = 4.4x GPU |
123+
| Baseline EPP | precise guide blend (prefix 3 / queue 2 / kv-util 2) |
124+
| P2P EPP | load-aware placement + P2P with load-aware source selection (llm-d-router#2032) |
125+
| Workload | 192 conversations, 16K system prompt, 8 turns (turn 8 arrives at ~48K input), 4K-token answers, 128 concurrent |
126+
127+
| Metric | Precise guide | Load + P2P | Delta |
128+
|---|---|---|---|
129+
| TTFT p50 | 2.53 s | 1.42 s | **-44%** |
130+
| TTFT p90 | 7.56 s | 4.87 s | **-36%** |
131+
| TTFT p99 | 25.4 s | 9.2 s | **-64%** |
132+
| Turn completion p90 (all turns) | 84.5 s | 78.3 s | -7% |
133+
| Throughput | 1.86 turns/s | 1.90 turns/s | +2% |
134+
135+
**Win:** an evicted or displaced session pays a full ~50K-token recompute
136+
before its first token under affinity (the 25.4 s p99); the pull fetches the
137+
conversation history instead (32.5M tokens moved), so replies start 1.8x
138+
sooner at median and the tail caps at 9.2 s. The short-answer variant (512-token
139+
answers, 256 concurrent) repeats the shape: TTFT p50 -23..-43%, p99 -40..-67%.
140+
141+
### 3.3 gpt-oss-120b — Uniform prefix pool (pull-value isolation)
142+
143+
| Setup | |
144+
|---|---|
145+
| Rig | 16x TP=1 (as 3.1) |
146+
| Arms | identical load-aware placement both sides; only difference is recompute vs pull on a cross-pod miss |
147+
| Workload | 128 shared prefixes x 48K (~6M-token working set = 4.4x one pod's GPU cache), 256-token questions, 64-token outputs, constant-rate stages |
148+
149+
| Metric | Load, no P2P | Load + P2P | Delta |
150+
|---|---|---|---|
151+
| Sustained rate (saturation) | 9.4 req/s | 16.7 req/s | **+78%** |
152+
| Req latency p50 @ 12 req/s | 24.9 s | 2.3 s | **-91%** |
153+
| Req latency p50 @ 16 req/s | 37.3 s | 3.6 s | **-90%** |
154+
| Decode throughput at peak | 599 tok/s | 1,068 tok/s | **+78%** |
155+
156+
**Win:** when the working set exceeds every pod's cache, each cross-pod miss
157+
costs a 1.7 s recompute; the pull converts it to a 0.55 s transfer (~139M
158+
prefix tokens pulled, ~58% of requests). This is the cleanest isolation of the
159+
pull itself — same placement in both arms.
160+
161+
### 3.4 Llama-3.1-8B — Shared-prefix pool and P/D
162+
163+
| Setup | |
164+
|---|---|
165+
| Model / pods | `meta-llama/Llama-3.1-8B-Instruct`, 4x TP=1, H200; P/D variant 4 prefill + 1 decode (NIXL between legs) |
166+
| KV / tiers | ~128 KB/token; GPU KV ~0.5M tokens/pod; CPU tier 32 GiB/pod |
167+
| Arms | load-aware placement both sides; +/- `p2p-source-producer` (`minCachedTokenDelta: 2048`) |
168+
| Workload | 64 shared prefixes x 16K (128 GiB KV pool, exceeds fleet cache) |
169+
170+
| Metric | Load, no P2P | Load + P2P | Delta |
171+
|---|---|---|---|
172+
| Req latency p50 @ 8 req/s | 2.49 s | 1.41 s | **-43%** |
173+
| Req latency p50 @ 12 req/s | 12.2 s | 2.1 s | **-83%** |
174+
| Saturation ceiling | 10.3 req/s | 12.6 req/s | **+22%** |
175+
| Peak token throughput | 2,420 tok/s | 3,184 tok/s | **+32%** |
176+
| P/D: achieved rate | 11.3 req/s | 14.7 req/s | **+30%** |
177+
| P/D: req latency p50 @ 16 req/s | 12.2 s | 5.6 s | **-54%** |
178+
179+
**Win:** same regime as 3.3 at quarter scale — and under P/D disaggregation
180+
the pull is what makes load-aware *prefill* placement viable at all: without
181+
it the prefill leg is recompute-bound (11.3 req/s); with it the topology
182+
returns to its decode-bound ceiling.
183+
184+
### 3.5 gpt-oss-120b — P/D disaggregation, docQA at C=192 (guide vs guide + P2P stack)
185+
186+
| Setup | |
187+
|---|---|
188+
| Rig | pd-disaggregation guide topology: 8 prefill + 8 decode, TP=1, 16x H200 |
189+
| Baseline | guide verbatim: plain `NixlConnector`, guide EPP |
190+
| P2P arm | identical + full stack: `MultiConnector(Nixl + Offloading[p2p tier])`, 128 GiB CPU tier both roles, sidecar `--enable-p2p-pull`, precise producer + `p2p-source-producer` (δ2048) |
191+
| Workload | docQA profile at concurrency 192 (mechanism-gated: C=192 produces placement spill, C=128 does not); clean re-roll per arm; 1,152/1,152 turns, zero failures both arms |
192+
193+
| Metric | Guide (plain NIXL) | Guide + P2P stack | Delta |
194+
|---|---|---|---|
195+
| TTFT p50 | 11.94 s | 1.16 s | **-90% (10x)** |
196+
| TTFT p95 | 71.6 s | 55.2 s | **-23%** |
197+
| TTFT p99 | 106.1 s | 80.0 s | **-25%** |
198+
| Throughput | 5.68 turns/s | 7.96 turns/s | **+40%** |
199+
200+
**Win (with honest attribution):** under 192-deep queues, turn N+1's history
201+
re-prefill hits the CPU offload tier (52.2M external-hit tokens) instead of
202+
recomputing — that tier is what buys the 10x median. The cross-pod pull fired
203+
zero times during this profile's smooth arrivals (it fired under the bursty
204+
mechanism gate at the same concurrency), so on P/D the pull rides free and
205+
activates under burstier traffic; the stack as a whole is what beats the
206+
shipping guide.
207+
208+
### 3.6 Qwen3-30B-A3B-Thinking — agentic multi-turn on P/D (guide vs guide + P2P)
209+
210+
| Setup | |
211+
|---|---|
212+
| Model / rig | `Qwen/Qwen3-30B-A3B-Thinking-2507` (agentic guide model), 2 prefill + 4 decode, TP=1, 6x H200; prefix caching on, `max-model-len` 131072 |
213+
| Baseline | agentic guide on plain NIXL |
214+
| P2P arm | + P2P stack; `minCachedTokenDelta: 1024`, calibrated on-rig (8K pull 74 ms vs 1.21 s recompute; crossover ~760 tokens) |
215+
| Workload | agentic synthetic: 24 conversations, 10K-100K-token system prompts (mean 50K), mean 12 turns, 1-20 s tool-call gaps (these evict session KV, making re-engagement the pull-vs-recompute choice), C=16, 288 requests; fresh fleet per arm |
216+
217+
| Metric | Guide (plain NIXL) | Guide + P2P (2 samples) | Delta |
218+
|---|---|---|---|
219+
| TTFT p50 | 5.22 s | 1.09 / 1.06 s | **-79% (4.8x)** |
220+
| TTFT p95 | 18.94 s | 11.77 / 14.79 s | **-38% / -22%** |
221+
| TTFT p99 | 30.29 s | 29.98 / 31.01 s | parity (cold 100K first-prefill floor) |
222+
| Run duration (288 reqs) | 304 s | 229 / 237 s | **-25% (+33% throughput)** |
223+
224+
**Win:** a returning agent whose session KV was evicted during a tool call
225+
pulls its history (1.23M tokens pulled in 229 s, every pull EPP-decided)
226+
instead of recomputing a ~50K context: replies start 4.8x sooner at median.
227+
The same prefill-from-decode mechanism validated on Llama-8B (Run N: per-turn
228+
TTFT flat at 0.1-0.2 s out to 20K-token prompts) ties there on latency because
229+
an 8B recompute is as cheap as the pull — the latency win appears when the
230+
recompute is expensive, exactly as the crossover predicts.
231+
232+
### 3.7 GLM-5.2-FP8 — precise routing + pull at agentic load
233+
234+
| Setup | |
235+
|---|---|
236+
| Model / pods | `zai-org/GLM-5.2-FP8` (753B MoE, MLA), 1 prefill + 1 decode, wide-EP DEP16 per role = 32x H200 |
237+
| KV / tiers | ~93 KB/token; GPU KV ~520K tokens/rank (prefill); CPU tier 100 GiB/rank; p2p secondary tier on port 7777 |
238+
| vLLM | `nightly-6a9f24aa` + `generic-p2p-src` overlay (unpaired cross-engine pull; upstream in-tree connector is paired-only until vllm#48021) |
239+
| Baseline EPP | armB: `precise-prefix-cache-producer` (kv-events) + prefix-affinity w5 / queue w3 / active w1 — no pull |
240+
| P2P EPP | armC-16k: armB + `p2p-source-producer` (`minCachedTokenDelta: 16384`) |
241+
| Workload | aiperf agentx (SemiAnalysis Weka agentic traces), concurrency 128, 900 s profiling, ~3.1-3.3K requests, 0 errors |
242+
243+
| Metric | armB (precise, no pull) | armC-16k (precise + pull) | Delta |
244+
|---|---|---|---|
245+
| TTFT p50 | 3,802 ms | 3,177 ms | **-16%** |
246+
| TTFT p90 | 11,755 ms | 9,970 ms | **-15%** |
247+
| TTFT p99 | 24,119 ms | 25,084 ms | +4% |
248+
| Output throughput | 2,016 tok/s | 1,982 tok/s | -2% |
249+
| KV pulled cross-engine | ~0 | 163 GB (1.77M tokens) | mechanism engaged |
250+
251+
Single-request mechanism proof on the same rig (24K-token prefix): cold
252+
recompute 4.63 s, cross-engine pull 3.91 s (**-16%**, 2.23 GB of KV moved
253+
per pull), warm local hit 2.30 s.
254+
255+
**Win:** first demonstration of the pull on a 753B wide-EP P/D deployment.
256+
Precise affinity concentrates agentic traffic on cache-holder ranks; the pull
257+
lets the picker route to a less-loaded rank and fetch the prefix, recovering
258+
16% median TTFT for ~2% throughput cost. Same-direction result as the gpt-oss
259+
conversational scenarios, on a model whose per-token KV is ~2x heavier.
260+
(Single run per arm; the c32/c64 precise cells are queued pending cluster
261+
GPU capacity.)
262+
263+
---
264+
265+
## 4. Where P2P does not help (measured, for completeness)
266+
267+
* **Cache-resident hot sets at steady state:** once every pod has acquired
268+
the hot prefixes, pull and recompute arms converge (gpt-oss scenario B,
269+
within 1-3%); the pull's value there is a one-time ~3x cheaper acquisition.
270+
* **Prefix-first placement that rarely misses:** adding P2P to the blend
271+
changed nothing (98K tokens moved all run vs 5.9M under load-aware) — the
272+
pull only helps a policy that generates misses by spreading.
273+
* **Same-prefix contention:** spreading traffic that *shares* the busy
274+
prefix converts free cache hits into a pull storm (scenario E); route to
275+
the cache in that case.
276+
* **Load-balanced routing on GLM agentic at saturation:** with approx
277+
routing already spreading load, adding the pull was neutral (within noise)
278+
at c32-c128 — the pull is miss-cost reduction, and good load balancing on
279+
this workload produced few expensive misses.
280+
* **The guide's own home workload (GPU-resident 6K prefixes):** placement to
281+
the cache wins outright; load-preferred placement + pull *loses* it (a pull
282+
per request: +80-100 ms p50, tails 5-16x at high rates) — never pay a pull
283+
for a free local hit. Adding the producer to the guide's own placement is
284+
**inert and free** (TTFT parity within 4 ms at every ladder stage), so it
285+
is safe as a default; the value claim belongs to the placement+pull pairing.
286+
* **KV-capacity-bound fleets (GPU KV squeezed 4.4x):** the pull is a no-op —
287+
it saves prefill compute but occupies the same KV for the same decode
288+
duration, and affinity additionally *deduplicates* KV that spreading forces
289+
per-pod. Boundary rule: P2P pays only when prefill compute/latency is the
290+
binding constraint and there is KV headroom to receive the copy.
291+
292+
One-line summary for the SIG: **P2P KV sharing converts a prefix-cache miss
293+
from "recompute" into "copy at a fraction of the cost" (-68% at 49K on
294+
gpt-oss, 16x on Qwen3-30B at 8K); measured value is largest where misses are
295+
frequent and expensive — working sets larger than any pod's cache (+78%
296+
throughput), conversational and agentic tails (2-4x p99 TTFT on gpt-oss, 4.8x
297+
median on Qwen3-30B P/D, 10x median for the P/D stack at C=192) — it fires on
298+
both precise and approximate indexes up to 753B wide-EP scale, and it is
299+
inert-and-free where it has nothing to do.**

0 commit comments

Comments
 (0)