P2P support

[liranschour]

P2P Connector — Feature Design

P2P Connector generalizes the PD Connector into a fully symmetric peer-to-peer mode. It reuses the same secondary-tier architecture (CPU KV cache in canonical layout, NIXL data path, ZMQ control path, unified bi-directional P2PSession) but drops the hard Prefiller / Decoder role distinction.

Every vLLM instance is a peer. For any given request, a peer plays one of two roles:

• Consumer (a.k.a. client): pulls KV blocks for the request from a remote peer's CPU cache instead of computing them locally.
• Producer (a.k.a. server): serves KV blocks from its CPU cache when a remote consumer asks for them.

A single peer can act as consumer for some requests and producer for others at the same time over the same session.

The orchestration layer indicates the request mode through kv_transfer_params. The same dict shape covers prefill/decode disaggregation and the symmetric P2P case.

Orchestration-Level Protocol

kv_transfer_params is a dict carrying one or more of the following keys. Each key's value is itself a dict with the fields required for that mode.

Each key names the remote counterpart this peer transfers with (not this peer's own role), so the name reads as "the remote ___ I transfer with".

Key	Set on	Value	Meaning
remote_decoder	Prefiller request	{kv_request_id: str}	This peer is the prefill producer for kv_request_id. Computed KV blocks must remain available for the remote decoder to pull.
remote_prefiller	Decoder request	{kv_request_id: str, remote_host: str, remote_port: int}	This peer is the decode consumer. Pull KV for this request from the remote prefiller at remote_host:remote_port, correlating via kv_request_id.
remote_kv_source	Consumer request	{kv_request_id: str, remote_host: str, remote_port: int}	This peer is a P2P consumer. Look up cached blocks at the remote source at remote_host:remote_port and pull whatever it has, correlating via kv_request_id.

Allowed combinations

The only legal multi-key combination is remote_decoder + remote_kv_source: a prefill producer is allowed to additionally act as a P2P consumer for the same request, looking up cached blocks at another peer to skip prefix prefill while still keeping its computed blocks available for a remote decoder to pull.

All other combinations are forbidden and must be rejected at the orchestration layer:

• remote_prefiller + remote_decoder — contradictory (a request cannot be both prefill producer and decode consumer).
• remote_prefiller + remote_kv_source — two competing fetch sources for the same request.
• All three keys together — forbidden by the two rules above.

Field semantics

• kv_request_id (str) — unique ID for the transfer transaction. In all modes the orchestrator allocates it and pushes it to the peer(s) involved. In PD it is set on both prefiller and decoder so they can correlate; in P2P it is set on the consumer and used on the wire when contacting the producer (the producer is implicit and serves whatever block hashes it currently holds in CPU cache).
• remote_host (str) — IP / hostname of the remote peer whose CPU cache will be queried.
• remote_port (int) — listening port on the remote peer.

Minimal Examples

# Prefiller request — keep KV available for a remote decoder to pull
kv_transfer_params = {
    "remote_decoder": {
        "kv_request_id": "<unique-transfer-id>",
    },
}

# Decoder request — pull KV from a specific prefiller
kv_transfer_params = {
    "remote_prefiller": {
        "kv_request_id": "<unique-transfer-id>",
        "remote_host": "<prefiller-ip>",
        "remote_port": <prefiller-port>,
    },
}

# P2P consumer request — pull whatever the source has cached
kv_transfer_params = {
    "remote_kv_source": {
        "kv_request_id": "<unique-transfer-id>",
        "remote_host": "<source-ip>",
        "remote_port": <source-port>,
    },
}

Sequence Diagram (P2P)

sequenceDiagram
    participant Cons_TM as Consumer TieringManager
    participant Cons_P2P as Consumer P2P Tier
    participant Prod_P2P as Producer P2P Tier
    participant Prod_TM as Producer TieringManager

    Cons_P2P->>Cons_P2P: Open listener thread
    Prod_P2P->>Prod_P2P: Open listener thread

    Note over Cons_TM,Prod_TM: Producer has previously cached blocks for this prompt

    Note over Cons_TM,Cons_P2P: Step N - aggregate per-block lookups

    Cons_TM->>Cons_P2P: lookup per block
    Note right of Cons_P2P: p2p mode<br/>Returns None and registers key

    Cons_TM->>Cons_P2P: on_schedule_end
    Cons_P2P->>Prod_P2P: 𝗖𝗧𝗥𝗟: LookupMsg kv_request_id block_hashes
    Note right of Cons_P2P: One msg per peer per request per step<br/>kv_request_id supplied by orchestrator via remote_kv_source

    Note right of Prod_P2P: Match hashes against local CPU cache

    Prod_P2P--)Cons_P2P: 𝗖𝗧𝗥𝗟: LookupRespMsg kv_request_id hit_block_hashes

    Note over Cons_TM,Cons_P2P: Step N+k - resolve

    Cons_TM->>Cons_P2P: lookup per block retry
    Note right of Cons_P2P: Hit returns True<br/>Miss returns False<br/>In-flight returns None

    Cons_TM->>Cons_P2P: submit_load hit blocks only
    Note right of Cons_P2P: CPU slots allocated for hits only

    Cons_P2P->>Prod_P2P: 𝗖𝗧𝗥𝗟: FetchMsg kv_request_id block_hashes dst_block_indexes

    Prod_P2P-)Cons_P2P: 𝗗𝗔𝗧𝗔: NIXL Transfer WRITE src_descs dst_descs
    Prod_P2P-->>Cons_P2P: 𝗖𝗧𝗥𝗟: TransferDone kv_request_id success

    Cons_TM->>Cons_P2P: get_finished
    Note right of Cons_P2P: Hits loaded into GPU as normal cache hit<br/>Misses recomputed by the engine

Loading

[elevran]

IIUC, there are 3 configurations sections that can be set under the kv_transfer_params

• p2p: { ... }: send to a consumer (the target that will prefetch) along with the indication of a single peer (IP and port) that can produce (the source). This is opportunistic, there is no guarantee how many blocks are available so the consumer needs to "lookup" to learn the producer state before fetching.
• prefill: { ... }: send to a prefiller in P/D disaggregation. Can be combined with p2p in case there is a good producer/source it may want to leverage before starting prefill work. This requires a kv_request_id that is the same as in decode section.
• decode: { ... }: send to a decoder in P/D disaggregation. Needs the same kv_request_id as in the corresponding prefill section sent.

Questions/comments:

• Is there a need for kv_request_id to be set and sent from the router in the p2p section? The consumer can generate a unique id on its own for "lookup" (or the producer can generate it in the lookup response)
• is there value in replacing the endpoint + port with a URL? In case we ever need to indicate the transport/transfer mechanism
• at least for p2p we may want to support one ore more targets. can leave a single entry for now, but we'll need to take care of backward compatibility when it materializes.