Skip to content

fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM #2140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vMaroon merged 4 commits into from
Jul 24, 2026
Merged

fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM #2140

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9a6ee75
fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM
nilig Jul 22, 2026
8b389d4
docs: wrap the renamed disaggregation paragraphs at 80 columns
nilig Jul 23, 2026
9cac78f
fix(sidecar): sweep stale p2p key references from comments and specs
nilig Jul 23, 2026
43aea48
refactor(sidecar): name request-field constants after their wire keys
nilig Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/disaggregation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,9 +539,9 @@ Specifies which KV transfer protocol the sidecar uses to coordinate prefill/deco
| `mooncake` | `MooncakeConnector` | [Mooncake](https://github.com/kvcache-ai/Mooncake) KV transfer using RDMA |
| `offloading` | `OffloadingConnector` | KV transfer over the vLLM CPU offloading tier. The decoder pulls KV from the prefiller via the `p2p` secondary tier. |

With `offloading`, the sidecar dispatches prefill and decode concurrently. It injects role-keyed `kv_transfer_params`: the prefiller receives `{"decode": {"kv_request_id": <id>}}` (no peer address), and the decoder receives `{"prefill": {"kv_request_id": <id>, "remote_host": <prefiller host>, "remote_port": <p2p-connector-port>}}` so it can pull KV from the prefiller. The prefiller host comes from the `x-prefiller-host-port` header; the port is `--p2p-connector-port`.
With `offloading`, the sidecar dispatches prefill and decode concurrently. It injects role-keyed `kv_transfer_params`, each key named for the remote party it describes: the prefiller receives `{"remote_decoder": {"kv_request_id": <id>}}` (no peer address), and the decoder receives `{"remote_prefiller": {"kv_request_id": <id>, "remote_host": <prefiller host>, "remote_port": <p2p-connector-port>}}` so it can pull KV from the prefiller. The prefiller host comes from the `x-prefiller-host-port` header; the port is `--p2p-connector-port`.

@shmuelk shmuelk Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While new lines in paragraphs as this one in Markdown files are meaningless, they make it much easier to review changes.

Please break this up into separate lines in the file.

@nilig nilig Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8b389d4.


When the request also carries the `x-kv-cache-source-host-port` header (set by the EPP `p2p-source-producer` to a peer holding more cached prefix than the pod computing the prefix), the sidecar injects an additional `p2p` key so vLLM pulls that cached prefix over the P2P tier instead of recomputing it. Under disaggregation the prefiller leg carries `{"decode": {...}, "p2p": {"kv_request_id": <own id>, "remote_host": <source host>, "remote_port": <p2p-connector-port>}}` (the only supported multi-key combination); without a prefiller the decoder-only request carries `{"p2p": {...}}` alone. A malformed or disallowed source header is ignored and the request proceeds unchanged, as is any source header on a connector that cannot pull over the P2P tier: only `offloading`, or NIXLv2 with `--enable-p2p-pull`, honors it. For the pulled blocks to be servable, the source pod must offload its generated (decode-phase) KV: set `offload_prompt_only: false` in its `kv_connector_extra_config` (the default `true` offloads only prefill blocks).
When the request also carries the `x-kv-cache-source-host-port` header (set by the EPP `p2p-source-producer` to a peer holding more cached prefix than the pod computing the prefix), the sidecar injects an additional `remote_kv_source` key so vLLM pulls that cached prefix over the P2P tier instead of recomputing it. Under disaggregation the prefiller leg carries `{"remote_decoder": {...}, "remote_kv_source": {"kv_request_id": <own id>, "remote_host": <source host>, "remote_port": <p2p-connector-port>}}` (the only supported multi-key combination); without a prefiller the decoder-only request carries `{"remote_kv_source": {...}}` alone. A malformed or disallowed source header is ignored and the request proceeds unchanged, as is any source header on a connector that cannot pull over the P2P tier: only `offloading`, or NIXLv2 with `--enable-p2p-pull`, honors it. For the pulled blocks to be servable, the source pod must offload its generated (decode-phase) KV: set `offload_prompt_only: false` in its `kv_connector_extra_config` (the default `true` offloads only prefill blocks).

@shmuelk shmuelk Jul 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto

@nilig nilig Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in 8b389d4.


Both prefill and decode pods require the following `--kv-transfer-config`:

Expand Down
6 changes: 3 additions & 3 deletions pkg/sidecar/proxy/connector_p2p.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,8 +232,8 @@ func (s *Server) addP2PPullToPrefill(prefillKVParams map[string]any, kvCacheSour
}
}

// p2pSourceParams builds the kv_transfer_params.p2p block for a pull from
// sourceHostPort's OffloadingConnector P2P tier. The kv_request_id is its
// p2pSourceParams builds the kv_transfer_params.remote_kv_source block for a
// pull from sourceHostPort's OffloadingConnector P2P tier. The kv_request_id is its
// own fresh UUID: in P2P mode it is consumer-side only.
Comment on lines +236 to 238

@nilig nilig Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The quoted comment was reworded in 9cac78f, and 43aea48 renames the requestFieldP2P* constants to match the wire keys. Remaining p2p references name the secondary-tier type, which is unchanged in vLLM.

func (s *Server) p2pSourceParams(sourceHostPort string) map[string]any {
return map[string]any{
Expand All @@ -244,7 +244,7 @@ func (s *Server) p2pSourceParams(sourceHostPort string) map[string]any {
}

// decodeWithP2PSource serves a decoder-only request through the local vLLM
// with kv_transfer_params.p2p injected, so the engine looks up and pulls
// with kv_transfer_params.remote_kv_source injected, so the engine looks up and pulls
// cached prefix blocks from the peer at sourceHostPort instead of recomputing
// them. It replaces any client-supplied kv_transfer_params (the sidecar owns
// that field) and routes through dispatchDecode so chunked decode still
Expand Down
4 changes: 2 additions & 2 deletions pkg/sidecar/proxy/connector_p2p_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ var _ = Describe("P2P Connector", func() {
return len(testInfo.prefillHandler.GetCompletionRequests())
}).Should(Equal(1))

// Prefill leg: kv_transfer_params.decode carries only kv_request_id,
// Prefill leg: kv_transfer_params.remote_decoder carries only kv_request_id,
// with no peer address.
prefillReqs := testInfo.prefillHandler.GetCompletionRequests()
Expect(prefillReqs).To(HaveLen(1))
Expand All @@ -82,7 +82,7 @@ var _ = Describe("P2P Connector", func() {
Expect(preq).To(HaveKeyWithValue(requestFieldMaxCompletionTokens, BeNumerically("==", 1)))
Expect(preq[requestFieldStream]).To(BeFalse())

// Decode leg: kv_transfer_params.prefill carries the prefiller's
// Decode leg: kv_transfer_params.remote_prefiller carries the prefiller's
// OffloadingConnector P2P tier address plus the matching kv_request_id.
Expect(testInfo.decodeHandler.RequestCount.Load()).To(BeNumerically("==", 1))
decodeReqs := testInfo.decodeHandler.GetCompletionRequests()
Expand Down
10 changes: 6 additions & 4 deletions pkg/sidecar/proxy/proxy.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,10 +92,12 @@ const (
requestFieldRemoteBootstrapAddr = "remote_bootstrap_addr"

// OffloadingConnector kv_transfer_params fields. The role is encoded by the
// nesting key: "decode" on the prefiller leg, "prefill" on the decoder leg.
requestFieldP2PDecodeParams = "decode"
requestFieldP2PPrefillParams = "prefill"
requestFieldP2PParams = "p2p"
// nesting key, named for the remote party it describes: "remote_decoder" on
// the prefiller leg, "remote_prefiller" on the decoder leg, "remote_kv_source"
// for a symmetric cached-prefix pull.
requestFieldP2PDecodeParams = "remote_decoder"
requestFieldP2PPrefillParams = "remote_prefiller"
requestFieldP2PParams = "remote_kv_source"
requestFieldKVRequestID = "kv_request_id"

KVConnectorNIXLV2 = constants.KVConnectorNIXLV2
Expand Down
2 changes: 1 addition & 1 deletion release-notes.d/unreleased/1937.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ url: https://github.com/llm-d/llm-d-router/pull/1937
author: nilig
date: 2026-07-14
---
The routing sidecar honors the `x-kv-cache-source-host-port` header, injecting `kv_transfer_params.p2p` so vLLM pulls cached prefix KV from the named peer over the OffloadingConnector P2P tier instead of recomputing it.
The routing sidecar honors the `x-kv-cache-source-host-port` header, injecting `kv_transfer_params.remote_kv_source` so vLLM pulls cached prefix KV from the named peer over the OffloadingConnector P2P tier instead of recomputing it.
Loading