p2p-findings: UC2 paired A/B does not reproduce the blog's Use Case 2

nilig · nilig · commit 91670a68e23c · 2026-07-27T07:43:36.000+03:00
Four counterbalanced ladders (A1 B1 B2 A2), fresh cold rig and
mechanism gate per run, arms differing by exactly one plugin. 22,080
requests, zero failures.

load+P2P is worth +2.0% achieved throughput at saturation, which is
smaller than the treatment arm's own run-to-run spread - individual
stages swing both directions across repeats, so any single-run headline
number is erased by its own repeat. Latency p50 is consistently better
for the pull at all four saturation rates (-13.2% to -1.2%), a real but
modest effect.

More importantly the premise does not reproduce: the no-pull control
does not collapse. It saturates cleanly at ~8.2 req/s across two full
ladders with zero failures, against a working set that genuinely
exceeds per-pod cache. Both arms hit the same prefill-bound ceiling.

Also records two silent methodology traps found while running this: a
ConfigMap-only helm upgrade does not restart the EPP, so the arm never
actually switches; and a gate prefix already cached fleet-wide cannot
demonstrate the pull, reading as a dead mechanism.

Signed-off-by: nilig &lt;nili.ifergan@gmail.com&gt;
diff --git a/test/p2p-findings/RESULTS-4.md b/test/p2p-findings/RESULTS-4.md
@@ -1,4 +1,10 @@
-# P2P benchmarks, series 4: UC2 lookup-hang resolved; UC3/UC4 re-validation
+# P2P benchmarks, series 4: UC2 lookup-hang resolved; UC3/UC4 re-validation; UC2 paired A/B
+
+> **Headline for the A/B section below**: on this rig and stack, `load+P2P`
+> does **not** reproduce the blog's Use Case 2 result. The pull is worth
+> about **+2% achieved throughput** at saturation — smaller than the
+> run-to-run spread — and the control arm (`load`, no pull) **does not
+> collapse**. See [UC2 paired A/B](#uc2-paired-ab-load-vs-loadp2p-4-ladders-counterbalanced).
 
 This series closes the lookup-hang blocker that stalled the blog's UC2/3/4
 re-validation ([[project_p2p_guide_docs_pr]], "BLOG UC2/3/4 RE-VALIDATION
@@ -298,3 +304,141 @@ re-derived tonight: Run N's actual throughput/latency comparison (TTFT
 parity verdict — "a capacity story, not latency, at this model scale") —
 that finding stands from the original investigation; this run only
 confirms the underlying mechanism hasn't regressed.
+
+---
+
+## UC2 paired A/B: `load` vs `load+P2P` (4 ladders, counterbalanced)
+
+The UC2 section above establishes only that the lookup-hang is fixed — it
+was a single-arm run. It does **not** test the blog's Use Case 2 *claim*,
+which is comparative. This section does, with a proper paired design.
+
+### Why this was needed
+
+The single-arm UC2 numbers landed suspiciously close to the `load`-no-pull
+baseline recorded 2026-07-25, which would contradict the blog. That
+comparison was invalid (different harness, different engine build), so it
+could not settle anything either way — but it was close enough to parity to
+require a real A/B before publishing behind the claim.
+
+### Design
+
+- **Arms differ by exactly one plugin.** `uc2-values-load.yaml` (control)
+  vs `uc2-values-load-p2p.yaml` (treatment); identical placement policy
+  (`queue-scorer` + `kv-cache-utilization-scorer` + `weighted-random-picker`),
+  the only delta being `p2p-source-producer`.
+- **Counterbalanced ABBA**: A1, B1, B2, A2 — so any monotonic cluster drift
+  cancels rather than loading onto one arm.
+- **Fresh cold rig per ladder**: all 4 engine pods deleted and recreated
+  before every run, so no arm inherits a warm cache from its predecessor.
+- **Mechanism gate before every ladder**, with a *unique* prefix per gate
+  (see trap 2 below). Treatment gates confirmed the exact pull signature —
+  1 owner pod at `load=0`, 3 non-owner pods each pulling 2.20 GB. Control
+  gates confirmed `load=0` on all 4 pods against a *fresh* prefix, so the
+  zero means "pull disabled", not "nothing left to pull".
+- Workload identical to `uc2_llama_pool.yaml.in`: 64-group shared-prefix
+  pool, 16K-token system prompt, 256 question/output tokens, stages
+  2/4/6/8/12/16/20/24 req/s x 60s. 5,520 requests per ladder, **22,080
+  total, zero failures across all four**.
+- Driver: [configs/uc2-ab-load-vs-loadp2p/uc2_ladder2.py](configs/uc2-ab-load-vs-loadp2p/uc2_ladder2.py),
+  which adds the metric the first pass lacked — **achieved throughput**
+  (completions / stage wall-clock including drain), the quantity the blog's
+  claim actually rests on.
+
+### Result: achieved throughput (req/s), saturation stages
+
+| rate | A1 | A2 | **A mean** | B1 | B2 | **B mean** | delta |
+|---:|---:|---:|---:|---:|---:|---:|---:|
+| 12 | 8.04 | 8.34 | **8.19** | 8.73 | 7.86 | **8.29** | +1.3% |
+| 16 | 8.30 | 8.25 | **8.28** | 8.09 | 8.41 | **8.25** | −0.3% |
+| 20 | 8.07 | 8.16 | **8.12** | 8.44 | 8.81 | **8.62** | +6.3% |
+| 24 | 8.06 | 8.22 | **8.14** | 8.56 | 7.88 | **8.22** | +1.0% |
+
+Overall saturation mean: **A = 8.18, B = 8.35 → +2.0% for the pull.**
+
+**That +2.0% is not distinguishable from noise.** The between-arm
+difference is 0.168 req/s; the within-arm run-to-run spread is 0.125
+(A) and 0.215 (B) req/s. The effect is smaller than the treatment arm's own
+variance. Individual stages swing both directions across repeats — rate 12
+went +8.6% in run 1 and −2.2% in run 2; rate 24 went +6.2% then −2.2%.
+Reporting any single stage would have produced a headline number that the
+repeat erases.
+
+### Result: latency p50, saturation stages
+
+| rate | A mean | B mean | delta |
+|---:|---:|---:|---:|
+| 12 | 26.0s | 22.6s | −13.2% |
+| 16 | 41.5s | 39.2s | −5.6% |
+| 20 | 51.2s | 50.6s | −1.2% |
+| 24 | 67.0s | 65.6s | −2.1% |
+
+Latency is the better-behaved signal: the pull is ahead at **all four**
+saturation rates. Direction is consistent (4/4), so a real effect is
+plausible, but the magnitude is small outside rate 12 and the design has
+n=2 per arm — enough to reject a large effect, not enough to size a small
+one precisely.
+
+### The premise does not reproduce
+
+The blog's UC2 narrative rests on load-balancing *without* the pull being
+catastrophic once the working set exceeds per-pod cache — the series-1
+observation that the no-pull arm "COLLAPSES", grinding to ~46s p50 at the
+top of the ladder.
+
+**That did not happen here.** The control arm saturated cleanly at ~8.2
+req/s and completed 11,040 requests across two full ladders with **zero
+failures** and zero restarts. It degrades with offered load exactly as a
+saturated system should, and the pull arm saturates at essentially the same
+ceiling. Both arms are prefill-capacity-bound at the same point; the pull
+shifts where time is spent, not how much total work the fleet can do.
+
+The working-set premise itself does hold on this rig (64 x 16K = ~1.02M
+tokens against a 16 GiB per-pod CPU tier, ~131K tokens at Llama-8B's ~128
+KB/token — so the pool genuinely does not fit per-pod). The regime is
+right; the dramatic outcome is not there.
+
+### What this does and does not license
+
+- It does **not** support publishing the blog's UC2 comparison as-is on
+  this stack.
+- It does **not** prove the pull is worthless — latency is consistently
+  better, and this is one workload shape at one scale on a 4-pod rig.
+  Series-1's collapse was observed on a different engine build with a
+  different harness; something real may have changed (the upstream fixes
+  altered exactly this path), or the original may have been measuring a
+  configuration that no longer exists.
+- It **does** mean the UC2 numbers need re-derivation before they back a
+  public claim, and that any re-derivation must be paired, counterbalanced,
+  and report achieved throughput with its run-to-run spread.
+
+### Two methodology traps found while running this
+
+Both are silent — they produce plausible-looking numbers rather than errors —
+and both would corrupt any A/B that hits them.
+
+1. **Switching EPP arms by editing the ConfigMap does not switch the arm.**
+   `helm upgrade` with only a `pluginsCustomConfig` change leaves the
+   Deployment podspec untouched, so no rollout fires, and the EPP reads its
+   plugin config once at startup. The first Arm B attempt ran with Arm A's
+   config still loaded (verified: zero occurrences of `p2p-source-producer`
+   in the running pod's log, pod creation timestamp predating the upgrade).
+   It looked like "the pull mechanism is dead". **An explicit
+   `kubectl rollout restart deploy/llm-d-router-epp` is required after every
+   arm switch**, and the loaded config should be asserted before trusting a
+   gate.
+2. **A gate prefix already cached on every pod cannot demonstrate the pull.**
+   Re-gating with the same prefix warms it everywhere; subsequent requests
+   are local hits, so `load` stays 0 and `store` stops growing. This is
+   indistinguishable from a broken mechanism by metrics alone. Gates must
+   use a **fresh, never-served prefix each time**
+   ([configs/uc2-ab-load-vs-loadp2p/gate_fresh.py](configs/uc2-ab-load-vs-loadp2p/gate_fresh.py)).
+   The tell that distinguishes the two cases: a frozen `store` counter means
+   "everything is already cached", a growing one means "computing locally
+   instead of pulling".
+
+Raw per-stage output for all four ladders:
+[ab_armA_run1.log](configs/uc2-ab-load-vs-loadp2p/ab_armA_run1.log),
+[ab_armB_run1.log](configs/uc2-ab-load-vs-loadp2p/ab_armB_run1.log),
+[ab_armB_run2.log](configs/uc2-ab-load-vs-loadp2p/ab_armB_run2.log),
+[ab_armA_run2.log](configs/uc2-ab-load-vs-loadp2p/ab_armA_run2.log).
diff --git a/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/ab_armA_run1.log b/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/ab_armA_run1.log
@@ -0,0 +1,10 @@
+# arm=armA-load-nopull-run1 endpoint=http://localhost:18090
+ rate   sent     ok  fail   wall_s  achieved   p50_ms   p95_ms
+    2    120    120     0     61.6      1.95     2864     4266
+    4    240    240     0     62.5      3.84     3031     5039
+    6    360    360     0     62.9      5.73     4804     9373
+    8    480    480     0     64.5      7.44    11008    16776
+   12    720    720     0     89.6      8.04    26099    35114
+   16    960    960     0    115.7      8.30    40475    57135
+   20   1200   1200     0    148.8      8.07    49907    84931
+   24   1440   1440     0    178.7      8.06    67598   115042
diff --git a/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/ab_armA_run2.log b/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/ab_armA_run2.log
@@ -0,0 +1,10 @@
+# arm=armA-load-nopull-run2 endpoint=http://localhost:18090
+ rate   sent     ok  fail   wall_s  achieved   p50_ms   p95_ms
+    2    120    120     0     61.9      1.94     2672     4081
+    4    240    240     0     63.0      3.81     3481     6465
+    6    360    360     0     62.8      5.73     4481     9526
+    8    480    480     0     65.3      7.35     9863    14846
+   12    720    720     0     86.4      8.34    25946    33424
+   16    960    960     0    116.3      8.25    42461    59775
+   20   1200   1200     0    147.1      8.16    52463    87269
+   24   1440   1440     0    175.2      8.22    66414   112482
diff --git a/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/ab_armB_run1.log b/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/ab_armB_run1.log
@@ -0,0 +1,10 @@
+# arm=armB-load-p2p-run1 endpoint=http://localhost:18090
+ rate   sent     ok  fail   wall_s  achieved   p50_ms   p95_ms
+    2    120    120     0     62.8      1.91     4101     6215
+    4    240    240     0     62.3      3.85     3393     5776
+    6    360    360     0     63.9      5.64     5618     9426
+    8    480    480     0     64.5      7.44     9685    14837
+   12    720    720     0     82.4      8.73    20552    29434
+   16    960    960     0    118.7      8.09    41626    58129
+   20   1200   1200     0    142.2      8.44    50630    82616
+   24   1440   1440     0    168.2      8.56    64595   107870
diff --git a/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/ab_armB_run2.log b/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/ab_armB_run2.log
@@ -0,0 +1,10 @@
+# arm=armB-load-p2p-run2 endpoint=http://localhost:18090
+ rate   sent     ok  fail   wall_s  achieved   p50_ms   p95_ms
+    2    120    120     0     62.0      1.94     2856     4217
+    4    240    240     0     63.0      3.81     3349     5435
+    6    360    360     0     63.0      5.71     4848     8466
+    8    480    480     0     65.0      7.38    11171    16038
+   12    720    720     0     91.7      7.86    24641    34601
+   16    960    960     0    114.2      8.41    36701    56691
+   20   1200   1200     0    136.2      8.81    50542    77456
+   24   1440   1440     0    182.7      7.88    66587   114357
diff --git a/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/gate_fresh.py b/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/gate_fresh.py
@@ -0,0 +1,17 @@
+import json, sys, time, urllib.request
+from concurrent.futures import ThreadPoolExecutor
+EP, NONCE = sys.argv[1], sys.argv[2]
+W = "route cache block prefix decode tier pull peer session lookup offload tensor page score filter epoch batch stream token merge".split()
+PREFIX = f"gate-{NONCE} unique pool document: " + " ".join(W[(hash(NONCE)+i) % len(W)] for i in range(16000))
+def one(i):
+    body = json.dumps({"model":"meta-llama/Llama-3.1-8B-Instruct","prompt":PREFIX+f" q{i}?","max_tokens":8,"temperature":0}).encode()
+    r = urllib.request.Request(EP+"/v1/completions", data=body, headers={"Content-Type":"application/json"})
+    try:
+        with urllib.request.urlopen(r, timeout=90) as resp:
+            return bool(json.loads(resp.read()).get("choices"))
+    except Exception as e:
+        return str(e)[:60]
+print("seed:", one(0)); time.sleep(12)
+with ThreadPoolExecutor(max_workers=8) as ex:
+    res = list(ex.map(one, range(1, 41)))
+print("burst ok:", sum(1 for x in res if x is True), "/ 40")
diff --git a/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/uc2-llama-resume.yaml b/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/uc2-llama-resume.yaml
@@ -0,0 +1,142 @@
+---
+# UC2 lookup-hang ladder re-test: #48021(merged, base image predates it so
+# fully overlaid) + #49877 (round-scoping fix for #49820) + #49850
+# (HIT_PENDING deadline fix for #49829, its scheduler.py already carries
+# #49671's merged crash fix -- confirmed via finished_signaled marker diff
+# against the live p2p-fix-49671 ConfigMap, so that older ConfigMap is
+# dropped). No defect12 reimplementation: the disconnect-recovery chain
+# (ClientRole.close -> failed_req_ids -> manager._failed_req_ids -> next
+# lookup() returns MISS) was verified directly against this code and the
+# duplicate-fetch trigger that caused it is what #49877 removes. This run
+# is the empirical confirmation of whether the 16-24 req/s hang recurs.
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: uc2-llama
+  namespace: nilig-p2p
+spec:
+  replicas: 4
+  selector:
+    matchLabels:
+      app: uc2-llama
+  template:
+    metadata:
+      labels:
+        app: uc2-llama
+        llm-d.ai/role: decode
+        llm-d.ai/inferenceServing: "true"
+        llm-d.ai/uc: uc2
+    spec:
+      initContainers:
+      - name: routing-proxy
+        image: quay.io/niliguy/llm-d-router-disagg-sidecar:p2p-rename-43aea48e
+        args:
+        - --port=8000
+        - --vllm-port=8200
+        - --kv-connector=offloading
+        - --p2p-connector-port=7777
+        - --secure-proxy=false
+        env:
+        - name: POD_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+        ports:
+        - containerPort: 8000
+          name: proxy
+        restartPolicy: Always
+        securityContext:
+          allowPrivilegeEscalation: false
+          runAsNonRoot: true
+      containers:
+      - name: modelserver
+        image: vllm/vllm-openai:nightly-1240c74c0a47473449cf0c3a9c2d87a1e159f73b
+        command: ["sh", "-c", "rm -f /dev/shm/vllm_offload_*.mmap 2>/dev/null; exec vllm serve \"$@\"", "sh"]
+        args:
+        - meta-llama/Llama-3.1-8B-Instruct
+        - --port=8200
+        - --disable-access-log-for-endpoints=/health,/metrics,/v1/models
+        - --tensor-parallel-size=1
+        - --block-size=64
+        - --max-model-len=20000
+        - --gpu-memory-utilization=0.6
+        - --kv-transfer-config
+        - '{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":17179869184,"offload_prompt_only":false,"secondary_tiers":[{"type":"p2p","host":"$(POD_IP)","port":7777}]}}'
+        - --kv-events-config
+        - '{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"$(KV_EVENTS_ENDPOINT)","topic":"kv@$(POD_IP):$(POD_PORT)@meta-llama/Llama-3.1-8B-Instruct"}'
+        env:
+        - name: HF_TOKEN
+          valueFrom:
+            secretKeyRef:
+              name: llm-d-hf-token
+              key: HF_TOKEN
+        - name: POD_IP
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+        - name: VLLM_P2P_SIDE_CHANNEL_HOST
+          valueFrom:
+            fieldRef:
+              fieldPath: status.podIP
+        - name: PYTHONHASHSEED
+          value: "0"
+        - name: KV_EVENTS_ENDPOINT
+          value: "tcp://*:5556"
+        - name: POD_PORT
+          value: "8000"
+        - name: HOME
+          value: /tmp
+        - name: VLLM_LOGGING_LEVEL
+          value: DEBUG
+        ports:
+        - containerPort: 8200
+          name: http
+        - containerPort: 7777
+          name: p2p
+        - containerPort: 5556
+          name: kv-events
+        resources:
+          limits:
+            cpu: "8"
+            memory: 64Gi
+            nvidia.com/gpu: "1"
+          requests:
+            cpu: "8"
+            memory: 64Gi
+            nvidia.com/gpu: "1"
+        volumeMounts:
+        - name: dshm
+          mountPath: /dev/shm
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/base.py, subPath: base.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/manager.py, subPath: manager.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/__init__.py, subPath: session___init__.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/client.py, subPath: session_client.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/protocol.py, subPath: session_protocol.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/server.py, subPath: session_server.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/session.py, subPath: session_session.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/control/zmq.py, subPath: control_zmq.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/control/base.py, subPath: control_base.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/offloading/scheduler.py, subPath: offloading_scheduler.py }
+        - { name: gp2p, mountPath: /usr/local/lib/python3.12/dist-packages/vllm/distributed/kv_transfer/kv_connector/v1/offloading/metrics.py, subPath: offloading_metrics.py }
+        readinessProbe:
+          httpGet: { path: /health, port: 8200 }
+          initialDelaySeconds: 30
+          periodSeconds: 10
+          failureThreshold: 60
+      volumes:
+      - name: dshm
+        emptyDir: { medium: Memory, sizeLimit: 24Gi }
+      - name: gp2p
+        configMap: { name: combined-overlay-uc2resume }
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: uc2-llama
+  namespace: nilig-p2p
+spec:
+  selector:
+    app: uc2-llama
+  ports:
+  - { name: proxy, port: 8000, targetPort: 8000 }
+  - { name: kv-events, port: 5556, targetPort: 5556 }
diff --git a/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/uc2-values-load-p2p.yaml b/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/uc2-values-load-p2p.yaml
diff --git a/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/uc2-values-load.yaml b/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/uc2-values-load.yaml
diff --git a/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/uc2_ladder2.py b/test/p2p-findings/configs/uc2-ab-load-vs-loadp2p/uc2_ladder2.py
