P2P KV cache sharing support in the router

[nilig]

Tracking issue for router support of the P2P Connector: any vLLM instance can pull cached prefix KV blocks from a peer's CPU cache instead of recomputing them, with the router deciding when and from whom.

Design: #1574 — defines the connector and the orchestration-level protocol (kv_transfer_params role sub-dicts, since renamed for clarity: remote_decoder / remote_prefiller / remote_kv_source, with remote_decoder + remote_kv_source the only legal combination; see #2133). The vLLM-side implementation lands via vllm-project/vllm#47636.

Router work

• EPP: p2p-source-producer emits x-kv-cache-source-host-port with the peer holding the most cached prefix tokens (EPP: add p2p-source-producer to emit the KV cache source header #1923, feat(epp): add p2p-source-producer for P2P prefix pulls #1924)
• Sidecar: consume x-kv-cache-source-host-port and inject the source-pull kv_transfer_params block (Sidecar: inject kv_transfer_params.p2p from the KV cache source header #1930; key later renamed p2p -> remote_kv_source, Sidecar P2P kv_transfer_params sub-dict keys diverged from the vLLM connector branch - pulls silently inert #2133)
• Sidecar: follow the connector branch's kv_transfer_params key rename (remote_prefiller / remote_decoder / remote_kv_source) - Sidecar P2P kv_transfer_params sub-dict keys diverged from the vLLM connector branch - pulls silently inert #2133, fix in fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM #2140
• Guide: P2P deployment overlay and documentation updates once the vLLM release containing the connector ships
• End-to-end validation: EPP emits the header -> sidecar injects the source-pull block -> vLLM pulls the blocks (peer logs write_blocks, puller reports kv_offload_load_bytes > 0). Validated twice against the connector dev branch (pull-proof: 1,216 tokens / 152 MB, kv_offload_load_bytes byte-exact for Llama-8B KV); remains open pending a vLLM release containing the connector.

Dependencies

• P2P support #1574 (design) and [KVOffload][P2P] Well-known default host/port env vars and per-DP-rank control port vllm-project/vllm#47636 (P2P tier: per-rank ports, identity fixes, load-failure propagation)
• PYTHONHASHSEED must be pinned identically across peers for cross-instance block-hash matching; deployment overlays must set it

[rahulgurnani]

How does this contrast with kv cache offloading to ssd/disk in write through mode? Since ssd/disk is shared across different pods, we may get similar performance. Ofcourse, for deployments when we don't have ssd/disk, P2P KV cache would be helpful.

[sagearc]

@rahulgurnani fair point. I think the distinction is less about P2P vs SSD specifically and more about direct-peer transfer vs a shared-tier design.

In a shared tier like mooncake store, someone has to track the state. Following that design means deploying a dedicated master node that tracks where all blocks live, and clients query it at lookup.

If we want a shared tier without an external master, the engines have to track who holds what themselves via kv events, which is valid but pushes coordination back into the engine and cuts against the efforts to keep state out of it (responses api, agentic api).

The suggestion here has the EPP track the state instead. That fits the flow, fewer network hops than the master option since the scheduler already plays that role, and it keeps the tracking external. It also sees kv cache from events as soon as a pod caches it, so it knows about a prefix even before it lands in the shared tier.