Skip to content

fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM #2140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vMaroon merged 4 commits into from
Jul 24, 2026
Merged

fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM #2140

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
9a6ee75
fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM
nilig Jul 22, 2026
8b389d4
docs: wrap the renamed disaggregation paragraphs at 80 columns
nilig Jul 23, 2026
9cac78f
fix(sidecar): sweep stale p2p key references from comments and specs
nilig Jul 23, 2026
43aea48
refactor(sidecar): name request-field constants after their wire keys
nilig Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 24 additions & 3 deletions docs/disaggregation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,9 +539,30 @@ Specifies which KV transfer protocol the sidecar uses to coordinate prefill/deco
| `mooncake` | `MooncakeConnector` | [Mooncake](https://github.com/kvcache-ai/Mooncake) KV transfer using RDMA |
| `offloading` | `OffloadingConnector` | KV transfer over the vLLM CPU offloading tier. The decoder pulls KV from the prefiller via the `p2p` secondary tier. |

With `offloading`, the sidecar dispatches prefill and decode concurrently. It injects role-keyed `kv_transfer_params`: the prefiller receives `{"decode": {"kv_request_id": <id>}}` (no peer address), and the decoder receives `{"prefill": {"kv_request_id": <id>, "remote_host": <prefiller host>, "remote_port": <p2p-connector-port>}}` so it can pull KV from the prefiller. The prefiller host comes from the `x-prefiller-host-port` header; the port is `--p2p-connector-port`.

When the request also carries the `x-kv-cache-source-host-port` header (set by the EPP `p2p-source-producer` to a peer holding more cached prefix than the pod computing the prefix), the sidecar injects an additional `p2p` key so vLLM pulls that cached prefix over the P2P tier instead of recomputing it. Under disaggregation the prefiller leg carries `{"decode": {...}, "p2p": {"kv_request_id": <own id>, "remote_host": <source host>, "remote_port": <p2p-connector-port>}}` (the only supported multi-key combination); without a prefiller the decoder-only request carries `{"p2p": {...}}` alone. A malformed or disallowed source header is ignored and the request proceeds unchanged, as is any source header on a connector that cannot pull over the P2P tier: only `offloading`, or NIXLv2 with `--enable-p2p-pull`, honors it. For the pulled blocks to be servable, the source pod must offload its generated (decode-phase) KV: set `offload_prompt_only: false` in its `kv_connector_extra_config` (the default `true` offloads only prefill blocks).
With `offloading`, the sidecar dispatches prefill and decode concurrently. It
injects role-keyed `kv_transfer_params`, each key named for the remote party it
describes: the prefiller receives `{"remote_decoder": {"kv_request_id": <id>}}`
(no peer address), and the decoder receives `{"remote_prefiller":
{"kv_request_id": <id>, "remote_host": <prefiller host>, "remote_port":
<p2p-connector-port>}}` so it can pull KV from the prefiller. The prefiller host
comes from the `x-prefiller-host-port` header; the port is
`--p2p-connector-port`.

When the request also carries the `x-kv-cache-source-host-port` header (set by
the EPP `p2p-source-producer` to a peer holding more cached prefix than the pod
computing the prefix), the sidecar injects an additional `remote_kv_source` key
so vLLM pulls that cached prefix over the P2P tier instead of recomputing it.
Under disaggregation the prefiller leg carries `{"remote_decoder": {...},
"remote_kv_source": {"kv_request_id": <own id>, "remote_host": <source host>,
"remote_port": <p2p-connector-port>}}` (the only supported multi-key
combination); without a prefiller the decoder-only request carries
`{"remote_kv_source": {...}}` alone. A malformed or disallowed source header is
ignored and the request proceeds unchanged, as is any source header on a
connector that cannot pull over the P2P tier: only `offloading`, or NIXLv2 with
`--enable-p2p-pull`, honors it. For the pulled blocks to be servable, the source
pod must offload its generated (decode-phase) KV: set `offload_prompt_only:
false` in its `kv_connector_extra_config` (the default `true` offloads only
prefill blocks).

Both prefill and decode pods require the following `--kv-transfer-config`:

Expand Down
4 changes: 2 additions & 2 deletions pkg/sidecar/proxy/connector_nixlv2_p2p_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ var _ = Describe("NIXL Connector with P2P pull", func() {
// NIXL fields still drive the NixlConnector under MultiConnector.
Expect(kv).To(HaveKeyWithValue(requestFieldDoRemoteDecode, true))
Expect(kv).To(HaveKeyWithValue(requestFieldDoRemotePrefill, false))
// The p2p block drives the OffloadingConnector's cached-prefix pull.
// The remote_kv_source block drives the OffloadingConnector's cached-prefix pull.
p2p, ok := kv[requestFieldP2PParams].(map[string]any)
Expect(ok).To(BeTrue())
Expect(p2p[requestFieldKVRequestID]).ToNot(BeEmpty())
Expand Down Expand Up @@ -157,7 +157,7 @@ var _ = Describe("NIXL Connector with P2P pull", func() {
body, _ := io.ReadAll(resp.Body) //nolint:errcheck
Expect(resp.StatusCode).To(Equal(http.StatusOK), string(body))

// Prefill leg keeps the NIXL WRITE fields and gains the composed p2p block.
// Prefill leg keeps the NIXL WRITE fields and gains the composed remote_kv_source block.
pkv := kvParams(env.prefillHandler, 0)
Expect(pkv).To(HaveKeyWithValue(requestFieldDoRemoteDecode, true))
p2p, ok := pkv[requestFieldP2PParams].(map[string]any)
Expand Down
17 changes: 9 additions & 8 deletions pkg/sidecar/proxy/connector_p2p.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -213,27 +213,28 @@ func (s *Server) handleP2PConcurrentRequests(w http.ResponseWriter, r *http.Requ
// KVConnector is offloading, or is composed alongside NIXL via MultiConnector
// (declared with --enable-p2p-pull) when the PD connector is NIXLv2. On any
// other connector --enable-p2p-pull has no effect, since no MultiConnector
// routes the p2p params to an OffloadingConnector.
// routes the remote_kv_source params to an OffloadingConnector.
func (s *Server) p2pPullAvailable() bool {
return s.config.KVConnector == KVConnectorOffloading ||
(s.config.EnableP2PPull && s.config.KVConnector == KVConnectorNIXLV2)
}

// addP2PPullToPrefill adds the OffloadingConnector p2p pull block to a prefill
// addP2PPullToPrefill adds the OffloadingConnector P2P pull block to a prefill

@roytman roytman Jul 23, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is some inconsistency in the names and comments P2P vs remote_kv_source, as in the comment https://github.com/llm-d/llm-d-router/pull/2140/changes#r3639185910

@nilig nilig Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed the constants after their wire keys in 43aea48: requestFieldRemoteDecoder, requestFieldRemotePrefiller, requestFieldRemoteKVSource, matching the sibling request-field constants, and swept the test comments that still said decode/prefill/p2p. P2P now only names the mechanism: the OffloadingConnector tier (type "p2p", unchanged in vLLM) and its protocol functions.

// leg's kv_transfer_params so the prefiller pulls cached prefix from
// kvCacheSource while keeping its own computed blocks available for the
// decoder. It is a no-op when no source is set or the source resolves to the
// prefiller itself, since there is nothing to pull from oneself. The p2p key
// composes with NIXL params: vLLM's MultiConnector routes it to the
// OffloadingConnector and the NIXL fields to the NixlConnector.
// prefiller itself, since there is nothing to pull from oneself. The
// remote_kv_source key composes with NIXL params: vLLM's MultiConnector
// routes it to the OffloadingConnector and the NIXL fields to the
// NixlConnector.
func (s *Server) addP2PPullToPrefill(prefillKVParams map[string]any, kvCacheSource, prefillPodHostPort string) {
if kvCacheSource != "" && extractHost(kvCacheSource) != extractHost(prefillPodHostPort) {
prefillKVParams[requestFieldP2PParams] = s.p2pSourceParams(kvCacheSource)
}
}

// p2pSourceParams builds the kv_transfer_params.p2p block for a pull from
// sourceHostPort's OffloadingConnector P2P tier. The kv_request_id is its
// p2pSourceParams builds the kv_transfer_params.remote_kv_source block for a
// pull from sourceHostPort's OffloadingConnector P2P tier. The kv_request_id is its
// own fresh UUID: in P2P mode it is consumer-side only.
Comment on lines +236 to 238

@nilig nilig Jul 23, 2026

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The quoted comment was reworded in 9cac78f, and 43aea48 renames the requestFieldP2P* constants to match the wire keys. Remaining p2p references name the secondary-tier type, which is unchanged in vLLM.

func (s *Server) p2pSourceParams(sourceHostPort string) map[string]any {
return map[string]any{
Expand All @@ -244,7 +245,7 @@ func (s *Server) p2pSourceParams(sourceHostPort string) map[string]any {
}

// decodeWithP2PSource serves a decoder-only request through the local vLLM
// with kv_transfer_params.p2p injected, so the engine looks up and pulls
// with kv_transfer_params.remote_kv_source injected, so the engine looks up and pulls
// cached prefix blocks from the peer at sourceHostPort instead of recomputing
// them. It replaces any client-supplied kv_transfer_params (the sidecar owns
// that field) and routes through dispatchDecode so chunked decode still
Expand Down
10 changes: 5 additions & 5 deletions pkg/sidecar/proxy/connector_p2p_source_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ var _ = Describe("P2P KV cache source header", func() {
return sendBody(proxyBaseAddr, chatCompletionsRequestBodyWithMaxCompletionTokens, headers)
}

It("should inject p2p params on the local request without disaggregation", func() {
It("should inject remote_kv_source params on the local request without disaggregation", func() {
proxyBaseAddr := testInfo.startProxy()

sendRequest(proxyBaseAddr, map[string]string{routing.KVCacheSourceHeader: kvCacheSource})
Expand All @@ -99,7 +99,7 @@ var _ = Describe("P2P KV cache source header", func() {
<-testInfo.stoppedCh
})

It("should add p2p params to the prefill leg under disaggregation", func() {
It("should add remote_kv_source params to the prefill leg under disaggregation", func() {
proxyBaseAddr := testInfo.startProxy()

prefillHostPort := testInfo.prefillBackend.URL[len("http://"):]
Expand Down Expand Up @@ -162,7 +162,7 @@ var _ = Describe("P2P KV cache source header", func() {
<-testInfo.stoppedCh
})

It("should not inject p2p params without the header", func() {
It("should not inject remote_kv_source params without the header", func() {
proxyBaseAddr := testInfo.startProxy()

sendRequest(proxyBaseAddr, nil)
Expand Down Expand Up @@ -198,7 +198,7 @@ var _ = Describe("P2P KV cache source header", func() {
Expect(decodeReqs).To(HaveLen(1))
kvParams, ok := decodeReqs[0][requestFieldKVTransferParams].(map[string]any)
Expect(ok).To(BeTrue())
// Only the sidecar-owned p2p key survives; the client's keys are gone.
// Only the sidecar-owned remote_kv_source key survives; the client's keys are gone.
Expect(kvParams).To(HaveLen(1))
p2p, ok := kvParams[requestFieldP2PParams].(map[string]any)
Expect(ok).To(BeTrue())
Expand All @@ -221,7 +221,7 @@ var _ = Describe("P2P KV cache source header", func() {
<-testInfo.stoppedCh
})

It("should not inject p2p params when the source is the local pod", func() {
It("should not inject remote_kv_source params when the source is the local pod", func() {
GinkgoT().Setenv("POD_IP", "10.9.9.9")
proxyBaseAddr := testInfo.startProxy()

Expand Down
4 changes: 2 additions & 2 deletions pkg/sidecar/proxy/connector_p2p_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ var _ = Describe("P2P Connector", func() {
return len(testInfo.prefillHandler.GetCompletionRequests())
}).Should(Equal(1))

// Prefill leg: kv_transfer_params.decode carries only kv_request_id,
// Prefill leg: kv_transfer_params.remote_decoder carries only kv_request_id,
// with no peer address.
prefillReqs := testInfo.prefillHandler.GetCompletionRequests()
Expect(prefillReqs).To(HaveLen(1))
Expand All @@ -82,7 +82,7 @@ var _ = Describe("P2P Connector", func() {
Expect(preq).To(HaveKeyWithValue(requestFieldMaxCompletionTokens, BeNumerically("==", 1)))
Expect(preq[requestFieldStream]).To(BeFalse())

// Decode leg: kv_transfer_params.prefill carries the prefiller's
// Decode leg: kv_transfer_params.remote_prefiller carries the prefiller's
// OffloadingConnector P2P tier address plus the matching kv_request_id.
Expect(testInfo.decodeHandler.RequestCount.Load()).To(BeNumerically("==", 1))
decodeReqs := testInfo.decodeHandler.GetCompletionRequests()
Expand Down
10 changes: 6 additions & 4 deletions pkg/sidecar/proxy/proxy.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -92,10 +92,12 @@ const (
requestFieldRemoteBootstrapAddr = "remote_bootstrap_addr"

// OffloadingConnector kv_transfer_params fields. The role is encoded by the
// nesting key: "decode" on the prefiller leg, "prefill" on the decoder leg.
requestFieldP2PDecodeParams = "decode"
requestFieldP2PPrefillParams = "prefill"
requestFieldP2PParams = "p2p"
// nesting key, named for the remote party it describes: "remote_decoder" on
// the prefiller leg, "remote_prefiller" on the decoder leg, "remote_kv_source"
// for a symmetric cached-prefix pull.
requestFieldP2PDecodeParams = "remote_decoder"
requestFieldP2PPrefillParams = "remote_prefiller"
requestFieldP2PParams = "remote_kv_source"
requestFieldKVRequestID = "kv_request_id"

KVConnectorNIXLV2 = constants.KVConnectorNIXLV2
Expand Down
2 changes: 1 addition & 1 deletion release-notes.d/unreleased/1937.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ url: https://github.com/llm-d/llm-d-router/pull/1937
author: nilig
date: 2026-07-14
---
The routing sidecar honors the `x-kv-cache-source-host-port` header, injecting `kv_transfer_params.p2p` so vLLM pulls cached prefix KV from the named peer over the OffloadingConnector P2P tier instead of recomputing it.
The routing sidecar honors the `x-kv-cache-source-host-port` header, injecting `kv_transfer_params.remote_kv_source` so vLLM pulls cached prefix KV from the named peer over the OffloadingConnector P2P tier instead of recomputing it.
Loading