Skip to content

Commit 263713e

Browse files
committed
docs(sidecar): restructure the DP section into plain-language bullets
One paragraph carried seven chained clauses; a lead sentence with one fact per bullet reads without unpacking. Signed-off-by: nilig <nili.ifergan@gmail.com>
1 parent a681071 commit 263713e

1 file changed

Lines changed: 22 additions & 1 deletion

File tree

docs/disaggregation.md

Lines changed: 22 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -559,7 +559,28 @@ Both prefill and decode pods require the following `--kv-transfer-config`:
559559

560560
`host` must be the pod's own IP at runtime (use the Kubernetes downward API env var `status.podIP`). `port` must match `--p2p-connector-port` (default `7777`). `cpu_bytes_to_use` controls the CPU KV offload buffer size; size it to hold the KV for the expected concurrent in-flight transfers. `OffloadingConnector` is available in vLLM nightly builds from 2026-06-30 onward (commit `bec232a`, [PR #42285](https://github.com/vllm-project/vllm/pull/42285)).
561561

562-
**Data parallelism:** with `--data-parallel-size` N > 1, vLLM binds DP replica `i`'s P2P tier on `<p2p-connector-port>+i`, where `i` is the **global** `data_parallel_index`, and gives each replica a distinct shared-memory offload region, so ranks in one pod do not collide. The sidecar derives the target's pod-local rank from the routed endpoint's port (rank `r` is served on `<port>+r`) and injects `remote_port` = `--p2p-connector-port` + `r`; a port outside the rank range falls back to rank 0. Local rank equals the global index only when each pod is its own DP group, so this derivation covers per-pod DP deployments; in multi-pod DP groups (e.g. LWS wide-EP, where pod k's replicas hold global indices k*N..k*N+N-1 behind the same serving ports) the source's global index must be supplied per request and is not derivable from the endpoint port. Requires vLLM with per-DP-rank P2P ports and per-replica offload regions ([PR #47636](https://github.com/vllm-project/vllm/pull/47636), [PR #47987](https://github.com/vllm-project/vllm/pull/47987)); on older engines every rank binds the same `POD_IP:<p2p-connector-port>` and DP > 1 fails at engine startup. Size `/dev/shm` for N regions: pod shm must exceed N x `cpu_bytes_to_use`.
562+
**Data parallelism:** the P2P tier supports `--data-parallel-size` N > 1 when
563+
each pod is a complete DP group (the per-pod DP deployment).
564+
565+
- vLLM gives each DP replica its own P2P listener and offload region:
566+
replica `i` serves on `<p2p-connector-port>+i`, where `i` is the
567+
**global** `data_parallel_index`
568+
([PR #47636](https://github.com/vllm-project/vllm/pull/47636),
569+
[PR #47987](https://github.com/vllm-project/vllm/pull/47987)). Engines
570+
without those changes bind every replica to the same
571+
`POD_IP:<p2p-connector-port>`, and DP > 1 fails at engine startup.
572+
- The sidecar serves rank `r` on its own port + `r`, so the routed
573+
endpoint's port names the target rank. The sidecar injects
574+
`remote_port` = `--p2p-connector-port` + `r`; a port outside the rank
575+
range falls back to rank 0.
576+
- The endpoint port encodes the pod-local rank, which matches the global
577+
index only when the pod is a whole DP group. In multi-pod DP groups (for
578+
example LWS wide-EP, where pod `k`'s replicas hold global indices
579+
`k*N..k*N+N-1` behind the same serving ports), the global index cannot
580+
be read from the port and would have to be supplied with the request, so
581+
those deployments are not covered.
582+
- Every replica maps its own offload region, so the pod's `/dev/shm` must
583+
exceed N x `cpu_bytes_to_use`.
563584

564585
### General Sidecar Flags
565586

0 commit comments

Comments
 (0)