findings(p2p): add Run N (Llama-8B EPP-e2e chat multi-turn P/D pulls)

nilig · nilig · commit 692c20a262cb · 2026-07-20T07:20:27.000+03:00
Run N was completed 2026-07-18 but never committed here like the other runs. It is the scale-out of Run M: 4P+4D and 2P+4D under the EPP on a concurrent chat workload, 477K tokens pulled at 4P/C=48 and 1.65M at 2P/C=96, per-turn TTFT flat at 0.1-0.2s while the prompt grows to ~20K. Honest caveat recorded: arm-A parity at this model scale (cheap recompute), so the value here is prefill capacity, not latency. Drivers and both EPP arms under configs/run-n.

Signed-off-by: nilig &lt;nili.ifergan@gmail.com&gt;
diff --git a/test/p2p-findings/RESULTS-2.md b/test/p2p-findings/RESULTS-2.md
@@ -337,6 +337,63 @@ prefill-from-decode works when the request can name decode's ids
 (kubernetes-sigs/inference-perf#649) or analysis-dropping templates
 (gpt-oss harmony) regardless of the serving stack.
 
+## Run N: EPP-e2e chat multi-turn, prefill pulls decode's history at scale (Llama-8B)
+
+Run M proved the cross-role transfer on a single request; Run N runs it
+end-to-end under the EPP, on a multi-turn chat workload at concurrency,
+across a P/D pair.
+
+Rig: `meta-llama/Llama-3.1-8B-Instruct`, 4 prefill + 4 decode (TP=1),
+then 2 prefill + 4 decode at saturation (`llama-pd.yaml`). Driver
+`chat_load.py`: concurrent conversations (48 at 4P, 96 at 2P), live
+history (each turn resends the accumulated chat), natural EOS (no
+`ignore_eos` - the same round-trip-stability requirement as Run M).
+Chat-completions rendered by the GPU engines (the CPU render image
+v0.23.0 lacks `/v1/chat/completions/render`;
+`llama-chat-render-svc.yaml`). Arms: A = precise prefix-cache placement,
+no pull (`epp-llama-a.yaml`); B = precise + `p2p-source-producer`,
+`minCachedTokenDelta: 1024` (`epp-llama-b.yaml`).
+
+Mechanism (arm B): EPP-decided per-turn prefill<-decode pulls fired on
+every turn where the scheduled prefill worker lacked the history. **477K
+tokens pulled (4P, C=48)** and **1.65M tokens (2P, C=96)**; the decode
+session accepted the pull on all decode pods. Per-turn TTFT stayed flat
+as the conversation grew (arm B):
+
+| turn | prompt (K tok) | TTFT p50 (s) | TTFT p95 (s) |
+|---|---|---|---|
+| 0 | 5.0  | 1.00 | 1.69 |
+| 1 | 7.1  | 0.10 | 0.12 |
+| 2 | 9.2  | 0.13 | 0.20 |
+| 3 | 11.3 | 0.15 | 0.17 |
+| 4 | 13.4 | 0.16 | 0.18 |
+| 5 | 15.5 | 0.18 | 0.21 |
+| 6 | 17.6 | 0.20 | 0.23 |
+| 7 | 19.7 | 0.21 | 0.23 |
+
+Turn 0 pays the cold prefill (1.0s p50); every later turn's history
+arrives by pull and TTFT holds at 0.1-0.2s even as the prompt grows to
+~20K tokens.
+
+BUT arm-A parity: on Llama-8B the recompute this replaces is cheap - a
+~2K-token answer re-prefills in ~60ms, about the pull's own cost - so arm
+A (recompute) and arm B (pull) tie on TTFT, and tails are dominated by
+decode first-token under batching, not prefill. Verdict at this model
+scale: the pull's value is prefill **capacity** (GPU-seconds saved), not
+user-visible latency. The latency win needs a regime where the recompute
+is expensive (longer history, slower/larger prefill, or prefill
+saturated) - which Run O (Qwen3-30B agentic) then demonstrates.
+
+Three config bugs cost hours (recorded so they do not recur): a
+token-heavy system prompt (~3.3 tok/word) blew `max-model-len`; a
+chat-render 404 from the old CPU render image; and a stale `modelName` in
+the token-producer config renders 404 silently and starves the whole
+producer chain (check `modelName` when cloning EPP configs).
+
+Configs: `configs/run-n` (`chat_load.py` driver, `epp-llama-a.yaml` /
+`epp-llama-b.yaml` arms, `llama-pd.yaml` rig,
+`llama-chat-render-svc.yaml`).
+
 ## Run O: agentic workload on P/D + P2P (Qwen3-30B-A3B-Thinking, EPP e2e)
 
 The agentic-serving guide's benchmark model and workload shapes, served on
diff --git a/test/p2p-findings/configs/run-n/chat_load.py b/test/p2p-findings/configs/run-n/chat_load.py
@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+"""Heavy-decode chat multi-turn load with live message history.
+
+Each conversation: unique ~6K-token system prompt, TURNS rounds; every round
+streams a long natural answer (temp 0, no ignore_eos) and appends it to the
+history, so turn N+1 carries the real generated text - the token-stable
+multi-turn that lets the EPP see decode-held KV. Reports per-turn TTFT
+(first SSE content chunk) and end-to-end times.
+
+usage: chat_load.py HOST PORT CONCURRENCY TURNS TAG OUTFILE
+"""
+import http.client, json, sys, threading, time
+
+HOST, PORT, C, TURNS, TAG, OUT = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]), int(sys.argv[4]), sys.argv[5], sys.argv[6]
+MODEL = "meta-llama/Llama-3.1-8B-Instruct"
+QS = ["Analyze the scenario in 15 detailed numbered sections.",
+      "Continue with 15 more sections, each a full paragraph.",
+      "Now give counterarguments for each point, in detail.",
+      "Propose alternatives, one paragraph each.",
+      "Assess risks of each alternative in detail.",
+      "Write an implementation plan with detailed steps.",
+      "Review the plan critically, section by section.",
+      "Summarize everything comprehensively section by section."]
+lock = threading.Lock()
+rows = []
+
+def turn(conn_host, msgs):
+    body = json.dumps({"model": MODEL, "messages": msgs, "max_tokens": 2500,
+                       "temperature": 0, "stream": True}).encode()
+    c = http.client.HTTPConnection(conn_host, PORT, timeout=600)
+    t0 = time.time(); ttft = None; text = []
+    c.request("POST", "/v1/chat/completions", body=body,
+              headers={"Content-Type": "application/json"})
+    r = c.getresponse()
+    if r.status != 200:
+        c.close(); raise RuntimeError(f"HTTP {r.status}: {r.read()[:120]}")
+    buf = b""
+    while True:
+        chunk = r.read1(65536)
+        if not chunk: break
+        buf += chunk
+        while b"\n" in buf:
+            line, buf = buf.split(b"\n", 1)
+            line = line.strip()
+            if not line.startswith(b"data:"): continue
+            payload = line[5:].strip()
+            if payload == b"[DONE]": continue
+            try: d = json.loads(payload)
+            except Exception: continue
+            delta = d.get("choices", [{}])[0].get("delta", {}).get("content")
+            if delta:
+                if ttft is None: ttft = time.time() - t0
+                text.append(delta)
+    c.close()
+    return "".join(text), ttft or (time.time() - t0), time.time() - t0
+
+def conv(i):
+    sysmsg = f"[{TAG}-c{i}] You are a meticulous analyst. " + " ".join(["ctx"] * 5000)
+    msgs = [{"role": "system", "content": sysmsg}]
+    for t in range(TURNS):
+        msgs.append({"role": "user", "content": QS[t % len(QS)]})
+        try:
+            a, ttft, e2e = turn(HOST, msgs)
+        except Exception as e:
+            with lock: rows.append({"conv": i, "turn": t, "err": str(e)[:80]})
+            return
+        msgs.append({"role": "assistant", "content": a})
+        with lock: rows.append({"conv": i, "turn": t, "ttft": round(ttft, 3), "e2e": round(e2e, 1), "chars": len(a)})
+
+t0 = time.time()
+ths = [threading.Thread(target=conv, args=(i,)) for i in range(C)]
+for x in ths: x.start()
+for x in ths: x.join()
+dur = time.time() - t0
+
+with open(OUT, "w") as f:
+    for r in rows: f.write(json.dumps(r) + "\n")
+ok = [r for r in rows if "ttft" in r]; er = [r for r in rows if "err" in r]
+def pct(v, p):
+    v = sorted(v); return v[min(len(v)-1, int(p*len(v)))] if v else 0
+print(f"{TAG}: turns ok={len(ok)} err={len(er)} dur={dur:.0f}s")
+for t in range(TURNS):
+    tv = [r["ttft"] for r in ok if r["turn"] == t]
+    if tv: print(f"  turn{t}: n={len(tv)} ttft p50={pct(tv,.5):.2f} p95={pct(tv,.95):.2f}")
+allt = [r["ttft"] for r in ok]
+print(f"  ALL: ttft p50={pct(allt,.5):.2f} p95={pct(allt,.95):.2f} p99={pct(allt,.99):.2f} gen_chars_avg={sum(r['chars'] for r in ok)//max(1,len(ok))}")
diff --git a/test/p2p-findings/configs/run-n/epp-llama-a.yaml b/test/p2p-findings/configs/run-n/epp-llama-a.yaml
@@ -0,0 +1,33 @@
+apiVersion: llm-d.ai/v1alpha1
+kind: EndpointPickerConfig
+# PD ARM A: the pd-disaggregation guide's shipped scheduling config, verbatim.
+# always-disagg, prefill profile prefix3/queue2/kv2, decode active2/prefix3.
+plugins:
+- type: disagg-headers-handler
+- type: always-disagg-pd-decider
+- type: disagg-profile-handler
+  parameters:
+    deciderPluginName: always-disagg-pd-decider
+- type: prefill-filter
+- type: decode-filter
+- type: prefix-cache-scorer
+- type: queue-scorer
+- type: kv-cache-utilization-scorer
+- type: active-request-scorer
+schedulingProfiles:
+- name: prefill
+  plugins:
+  - pluginRef: prefill-filter
+  - pluginRef: prefix-cache-scorer
+    weight: 3
+  - pluginRef: queue-scorer
+    weight: 2
+  - pluginRef: kv-cache-utilization-scorer
+    weight: 2
+- name: decode
+  plugins:
+  - pluginRef: decode-filter
+  - pluginRef: active-request-scorer
+    weight: 2
+  - pluginRef: prefix-cache-scorer
+    weight: 3
diff --git a/test/p2p-findings/configs/run-n/epp-llama-b.yaml b/test/p2p-findings/configs/run-n/epp-llama-b.yaml
@@ -0,0 +1,61 @@
+apiVersion: llm-d.ai/v1alpha1
+kind: EndpointPickerConfig
+# PD ARM B: ARM A + P2P only. Identical deciders, filters, scorers, weights.
+# Adds the machinery the pull needs: token-producer + precise index (feeds
+# ONLY the p2p-source-producer; the guide's prefix-cache-scorer keeps its
+# default approximate feed so placement is identical to arm A) and the
+# p2p-source-producer targeting the prefill profile.
+plugins:
+- type: disagg-headers-handler
+- type: always-disagg-pd-decider
+- type: disagg-profile-handler
+  parameters:
+    deciderPluginName: always-disagg-pd-decider
+- type: prefill-filter
+- type: decode-filter
+- type: token-producer
+  parameters:
+    modelName: meta-llama/Llama-3.1-8B-Instruct
+    vllm:
+      url: http://llama-chat-render:8000
+- type: endpoint-notification-source
+- type: precise-prefix-cache-producer
+  parameters:
+    tokenProcessorConfig:
+      blockSize: 64
+    kvEventsConfig:
+      topicFilter: "kv@"
+      discoverPods: true
+      podDiscoveryConfig:
+        socketPort: 5556
+- type: p2p-source-producer
+  parameters:
+    prefixMatchInfoProducerName: precise-prefix-cache-producer
+    prefillProfileName: prefill
+    minCachedTokenDelta: 1024
+- type: prefix-cache-scorer
+- type: queue-scorer
+- type: kv-cache-utilization-scorer
+- type: active-request-scorer
+dataLayer:
+  sources:
+  - pluginRef: endpoint-notification-source
+    extractors:
+    - pluginRef: precise-prefix-cache-producer
+schedulingProfiles:
+- name: prefill
+  plugins:
+  - pluginRef: prefill-filter
+  - pluginRef: prefix-cache-scorer
+    weight: 3
+  - pluginRef: queue-scorer
+    weight: 2
+  - pluginRef: kv-cache-utilization-scorer
+    weight: 2
+- name: decode
+  plugins:
+  - pluginRef: decode-filter
+  - pluginRef: active-request-scorer
+    weight: 2
+  - pluginRef: prefix-cache-scorer
+    weight: 3
diff --git a/test/p2p-findings/configs/run-n/llama-chat-render-svc.yaml b/test/p2p-findings/configs/run-n/llama-chat-render-svc.yaml
@@ -0,0 +1,6 @@
+apiVersion: v1
+kind: Service
+metadata: {name: llama-chat-render, namespace: nilig-p2p}
+spec:
+  selector: {app: llama-pd, role: prefill}
+  ports: [{port: 8000, targetPort: 8000}]
diff --git a/test/p2p-findings/configs/run-n/llama-pd.yaml b/test/p2p-findings/configs/run-n/llama-pd.yaml
@@ -0,0 +1,128 @@
+# DP-aware sidecar validation rig: 1 prefill (engine only) + 1 decode
+# (sidecar:dp-aware + engine). Llama-8B TP=1, MultiConnector(NIXL +
+# Offloading p2p tier), generic_p2p overlay. No EPP labels - driven by
+# direct curls with x-prefiller-host-port / x-kv-cache-source-host-port.
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: llama-pd-prefill, namespace: nilig-p2p, labels: {app: llama-pd, role: prefill}}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: llama-pd, role: prefill}}
+  template:
+    metadata: {labels: {app: llama-pd, role: prefill}}
+    spec:
+      serviceAccountName: pd-disaggregation-nvidia-gpu-vllm-sa
+      containers:
+        - name: modelserver
+          image: quay.io/niliguy/vllm-openai:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9
+          imagePullPolicy: IfNotPresent
+          command: ["sh","-c","rm -f /dev/shm/vllm_offload_*.mmap 2>/dev/null; exec vllm serve \"$@\"","sh"]
+          args:
+            - meta-llama/Llama-3.1-8B-Instruct
+            - --port=8000
+            - --tensor-parallel-size=1
+            - --block-size=64
+            - --max-model-len=20000
+            - --gpu-memory-utilization=0.6
+            - --kv-events-config
+            - '{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:5556","topic":"kv@$(POD_IP):8000@meta-llama/Llama-3.1-8B-Instruct"}'
+            - --kv-transfer-config
+            - '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both"},{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":17179869184,"offload_prompt_only":false,"secondary_tiers":[{"type":"p2p","host":"$(POD_IP)","port":7777}]}}]}}'
+          env:
+            - {name: HF_TOKEN, valueFrom: {secretKeyRef: {name: llm-d-hf-token, key: HF_TOKEN}}}
+            - {name: POD_IP, valueFrom: {fieldRef: {apiVersion: v1, fieldPath: status.podIP}}}
+            - {name: PYTHONHASHSEED, value: "0"}
+            - {name: VLLM_NIXL_SIDE_CHANNEL_HOST, valueFrom: {fieldRef: {apiVersion: v1, fieldPath: status.podIP}}}
+            - {name: VLLM_NIXL_SIDE_CHANNEL_PORT, value: "5600"}
+          ports: [{containerPort: 8000}, {containerPort: 5600}, {containerPort: 7777}, {containerPort: 5556}]
+          resources: {limits: {cpu: "8", memory: 48Gi, nvidia.com/gpu: "1", rdma/ib: "1"}, requests: {cpu: "8", memory: 48Gi, nvidia.com/gpu: "1", rdma/ib: "1"}}
+          startupProbe: {httpGet: {path: /v1/models, port: 8000}, initialDelaySeconds: 10, periodSeconds: 15, failureThreshold: 60}
+          readinessProbe: {httpGet: {path: /v1/models, port: 8000}, periodSeconds: 5, failureThreshold: 3}
+          volumeMounts:
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/envs.py, name: gp2p, subPath: envs.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/spec.py, name: gp2p, subPath: spec.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/manager.py, name: gp2p, subPath: tiering_manager.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/manager.py, name: gp2p, subPath: manager.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/data/base.py, name: gp2p, subPath: data_base.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/data/nixl.py, name: gp2p, subPath: data_nixl.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/client.py, name: gp2p, subPath: session_client.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/protocol.py, name: gp2p, subPath: session_protocol.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/server.py, name: gp2p, subPath: session_server.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/session.py, name: gp2p, subPath: session_session.py}
+            - {mountPath: /dev/shm, name: shm}
+            - {mountPath: /.cache, name: cache}
+            - {mountPath: /.config, name: cfg}
+            - {mountPath: /.triton, name: triton}
+      volumes:
+        - {name: shm, emptyDir: {medium: Memory, sizeLimit: 24Gi}}
+        - {name: gp2p, configMap: {name: generic-p2p-src}}
+        - {name: cache, emptyDir: {}}
+        - {name: cfg, emptyDir: {}}
+        - {name: triton, emptyDir: {}}
+---
+apiVersion: apps/v1
+kind: Deployment
+metadata: {name: llama-pd-decode, namespace: nilig-p2p, labels: {app: llama-pd, role: decode}}
+spec:
+  replicas: 1
+  selector: {matchLabels: {app: llama-pd, role: decode}}
+  template:
+    metadata: {labels: {app: llama-pd, role: decode}}
+    spec:
+      serviceAccountName: pd-disaggregation-nvidia-gpu-vllm-sa
+      initContainers:
+        - name: routing-proxy
+          image: quay.io/niliguy/llm-d-router-disagg-sidecar:dp-aware
+          imagePullPolicy: Always
+          restartPolicy: Always
+          args: ["--port=8000","--vllm-port=8200","--kv-connector=nixlv2","--enable-p2p-pull","--zap-log-level=2","--secure-proxy=false"]
+          env:
+            - {name: POD_IP, valueFrom: {fieldRef: {apiVersion: v1, fieldPath: status.podIP}}}
+          ports: [{containerPort: 8000}]
+      containers:
+        - name: modelserver
+          image: quay.io/niliguy/vllm-openai:nightly-2afa3f7e950264bb179d030c23a1ed1f46558fd9
+          imagePullPolicy: IfNotPresent
+          command: ["sh","-c","rm -f /dev/shm/vllm_offload_*.mmap 2>/dev/null; exec vllm serve \"$@\"","sh"]
+          args:
+            - meta-llama/Llama-3.1-8B-Instruct
+            - --port=8200
+            - --tensor-parallel-size=1
+            - --block-size=64
+            - --max-model-len=20000
+            - --gpu-memory-utilization=0.6
+            - --kv-events-config
+            - '{"enable_kv_cache_events":true,"publisher":"zmq","endpoint":"tcp://*:5556","topic":"kv@$(POD_IP):8000@meta-llama/Llama-3.1-8B-Instruct"}'
+            - --kv-transfer-config
+            - '{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both"},{"kv_connector":"OffloadingConnector","kv_role":"kv_both","kv_connector_extra_config":{"spec_name":"TieringOffloadingSpec","cpu_bytes_to_use":17179869184,"offload_prompt_only":false,"secondary_tiers":[{"type":"p2p","host":"$(POD_IP)","port":7777}]}}]}}'
+          env:
+            - {name: HF_TOKEN, valueFrom: {secretKeyRef: {name: llm-d-hf-token, key: HF_TOKEN}}}
+            - {name: POD_IP, valueFrom: {fieldRef: {apiVersion: v1, fieldPath: status.podIP}}}
+            - {name: PYTHONHASHSEED, value: "0"}
+            - {name: VLLM_NIXL_SIDE_CHANNEL_HOST, valueFrom: {fieldRef: {apiVersion: v1, fieldPath: status.podIP}}}
+            - {name: VLLM_NIXL_SIDE_CHANNEL_PORT, value: "5600"}
+          ports: [{containerPort: 8200}, {containerPort: 5600}, {containerPort: 7777}, {containerPort: 5556}]
+          resources: {limits: {cpu: "8", memory: 48Gi, nvidia.com/gpu: "1", rdma/ib: "1"}, requests: {cpu: "8", memory: 48Gi, nvidia.com/gpu: "1", rdma/ib: "1"}}
+          startupProbe: {httpGet: {path: /v1/models, port: 8200}, initialDelaySeconds: 10, periodSeconds: 15, failureThreshold: 60}
+          readinessProbe: {httpGet: {path: /v1/models, port: 8200}, periodSeconds: 5, failureThreshold: 3}
+          volumeMounts:
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/envs.py, name: gp2p, subPath: envs.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/spec.py, name: gp2p, subPath: spec.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/manager.py, name: gp2p, subPath: tiering_manager.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/manager.py, name: gp2p, subPath: manager.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/data/base.py, name: gp2p, subPath: data_base.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/data/nixl.py, name: gp2p, subPath: data_nixl.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/client.py, name: gp2p, subPath: session_client.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/protocol.py, name: gp2p, subPath: session_protocol.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/server.py, name: gp2p, subPath: session_server.py}
+            - {mountPath: /usr/local/lib/python3.12/dist-packages/vllm/v1/kv_offload/tiering/p2p/session/session.py, name: gp2p, subPath: session_session.py}
+            - {mountPath: /dev/shm, name: shm}
+            - {mountPath: /.cache, name: cache}
+            - {mountPath: /.config, name: cfg}
+            - {mountPath: /.triton, name: triton}
+      volumes:
+        - {name: shm, emptyDir: {medium: Memory, sizeLimit: 24Gi}}
+        - {name: gp2p, configMap: {name: generic-p2p-src}}
+        - {name: cache, emptyDir: {}}
+        - {name: cfg, emptyDir: {}}
+        - {name: triton, emptyDir: {}}
