fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM by nilig · Pull Request #2140 · llm-d/llm-d-router

nilig · 2026-07-22T13:09:55Z

What type of PR is this?
/kind bug

What this PR does / why we need it:
Renames the sidecar's P2P kv_transfer_params sub-dict keys to match the vLLM OffloadingConnector branch after liranschour/vllm@530fa0d7 + @5bcf5b31: prefill -> remote_prefiller, decode -> remote_decoder, p2p -> remote_kv_source, each named for the remote party it describes. Inner fields (kv_request_id, remote_host, remote_port) and the p2p secondary-tier type are unchanged. Without this, a sidecar paired with the current branch injects keys the engine ignores, so cached-prefix pulls are silently inert - requests serve, everything recomputes.

The three literals live only in the requestFieldP2P* constants, so the change is the constant values plus comments and docs/disaggregation.md; all injection sites and tests flow through the constants. Constant identifiers are kept to minimize churn. Sidecar and engine must move together: mixing old and new sides on either arrangement leaves pulls silently disabled (both engine generations ignore foreign keys, so nothing crashes).

Which issue(s) this PR fixes:
Fixes #2133

Release note:

The routing sidecar's P2P kv_transfer_params sub-dict keys are renamed to match the vLLM OffloadingConnector branch: "prefill" -> "remote_prefiller", "decode" -> "remote_decoder", "p2p" -> "remote_kv_source". Sidecar and engine must be upgraded together: mixing old and new sides leaves cached-prefix pulls silently disabled.

Test plan:

Sidecar unit suite through the renamed keys (113/113; assertions flow through the constants, exercising every injection path)
make lint
make presubmit up to vulncheck; vulncheck fails identically on clean main (pre-existing GO-2026-5970, unrelated x/text)
Cloud pull-proof against a branch-HEAD engine (overlay pinned at liranschour/vllm@8054ff13 (branch HEAD with the final names) over nightly-dcfebf93, Llama-8B 1P1D): with the source header the prefill worker pulled 1,216 tokens / 152 MB of the decoder's generated history (125 KB/token, byte-exact for Llama-8B bf16 KV); control without the header: no pull. Verified in-pod that the loaded engine parses remote_kv_source and no longer parses p2p.

The OffloadingConnector P2P branch renamed the request-level sub-dict keys for clarity (liranschour/vllm@530fa0d7 + @5bcf5b31): "prefill" -> "remote_prefiller", "decode" -> "remote_decoder", "p2p" -> "remote_kv_source", each named for the remote party it describes. Inner fields (kv_request_id, remote_host, remote_port) and the secondary-tier type "p2p" are unchanged. A sidecar emitting the old keys against a renamed engine is silently inert - the engine ignores unknown sub-dicts and every request recomputes - so the sidecar and engine must move together. Signed-off-by: nilig <nili.ifergan@gmail.com>

shmuelk · 2026-07-22T13:52:19Z

 | `offloading` | `OffloadingConnector` | KV transfer over the vLLM CPU offloading tier. The decoder pulls KV from the prefiller via the `p2p` secondary tier. |

-With `offloading`, the sidecar dispatches prefill and decode concurrently. It injects role-keyed `kv_transfer_params`: the prefiller receives `{"decode": {"kv_request_id": <id>}}` (no peer address), and the decoder receives `{"prefill": {"kv_request_id": <id>, "remote_host": <prefiller host>, "remote_port": <p2p-connector-port>}}` so it can pull KV from the prefiller. The prefiller host comes from the `x-prefiller-host-port` header; the port is `--p2p-connector-port`.
+With `offloading`, the sidecar dispatches prefill and decode concurrently. It injects role-keyed `kv_transfer_params`, each key named for the remote party it describes: the prefiller receives `{"remote_decoder": {"kv_request_id": <id>}}` (no peer address), and the decoder receives `{"remote_prefiller": {"kv_request_id": <id>, "remote_host": <prefiller host>, "remote_port": <p2p-connector-port>}}` so it can pull KV from the prefiller. The prefiller host comes from the `x-prefiller-host-port` header; the port is `--p2p-connector-port`.


While new lines in paragraphs as this one in Markdown files are meaningless, they make it much easier to review changes.

Please break this up into separate lines in the file.

Done in 8b389d4.

shmuelk · 2026-07-22T13:52:32Z

+With `offloading`, the sidecar dispatches prefill and decode concurrently. It injects role-keyed `kv_transfer_params`, each key named for the remote party it describes: the prefiller receives `{"remote_decoder": {"kv_request_id": <id>}}` (no peer address), and the decoder receives `{"remote_prefiller": {"kv_request_id": <id>, "remote_host": <prefiller host>, "remote_port": <p2p-connector-port>}}` so it can pull KV from the prefiller. The prefiller host comes from the `x-prefiller-host-port` header; the port is `--p2p-connector-port`.

-When the request also carries the `x-kv-cache-source-host-port` header (set by the EPP `p2p-source-producer` to a peer holding more cached prefix than the pod computing the prefix), the sidecar injects an additional `p2p` key so vLLM pulls that cached prefix over the P2P tier instead of recomputing it. Under disaggregation the prefiller leg carries `{"decode": {...}, "p2p": {"kv_request_id": <own id>, "remote_host": <source host>, "remote_port": <p2p-connector-port>}}` (the only supported multi-key combination); without a prefiller the decoder-only request carries `{"p2p": {...}}` alone. A malformed or disallowed source header is ignored and the request proceeds unchanged, as is any source header on a connector that cannot pull over the P2P tier: only `offloading`, or NIXLv2 with `--enable-p2p-pull`, honors it. For the pulled blocks to be servable, the source pod must offload its generated (decode-phase) KV: set `offload_prompt_only: false` in its `kv_connector_extra_config` (the default `true` offloads only prefill blocks).
+When the request also carries the `x-kv-cache-source-host-port` header (set by the EPP `p2p-source-producer` to a peer holding more cached prefix than the pod computing the prefix), the sidecar injects an additional `remote_kv_source` key so vLLM pulls that cached prefix over the P2P tier instead of recomputing it. Under disaggregation the prefiller leg carries `{"remote_decoder": {...}, "remote_kv_source": {"kv_request_id": <own id>, "remote_host": <source host>, "remote_port": <p2p-connector-port>}}` (the only supported multi-key combination); without a prefiller the decoder-only request carries `{"remote_kv_source": {...}}` alone. A malformed or disallowed source header is ignored and the request proceeds unchanged, as is any source header on a connector that cannot pull over the P2P tier: only `offloading`, or NIXLv2 with `--enable-p2p-pull`, honors it. For the pulled blocks to be servable, the source pod must offload its generated (decode-phase) KV: set `offload_prompt_only: false` in its `kv_connector_extra_config` (the default `true` offloads only prefill blocks).


Done in 8b389d4.

shmuelk · 2026-07-22T13:54:33Z

@nilig While the vLLM PR that changed the KV related JSON fileds has been merged. Is there is a version of vLLM available with the PR? Having users needing to build vLLM from source is not a good idea...

Review feedback on llm-d#2140: the two protocol paragraphs were single long lines; wrapped for reviewable diffs. No content change. Signed-off-by: nilig <nili.ifergan@gmail.com>

nilig · 2026-07-23T11:05:41Z

The vLLM side is not merged yet - the rename lives in vllm-project/vllm#48021, which is approved (orozery) and carries the ready label, so merge looks imminent. No source builds either way: once it merges, the public nightly images (vllm/vllm-openai:nightly-<sha>) carry the connector, and that image is what the guide deploys; a tagged release follows the normal cadence.

Sequencing-wise I'd merge this PR together with or after the vLLM merge. Until then the guide pins engine and sidecar as a matched pair, and mixing old/new sides degrades to recompute rather than failing, so there is no crash window - just the silently-inert pairing this PR exists to prevent.

Copilot

Pull request overview

This PR updates the sidecar’s OffloadingConnector P2P kv_transfer_params sub-dict key names to match the current vLLM OffloadingConnector branch, ensuring cached-prefix pulls are actually recognized by the engine instead of being silently ignored.

Changes:

Renamed kv_transfer_params nesting keys emitted by the sidecar to remote_decoder, remote_prefiller, and remote_kv_source (via the existing requestFieldP2P* constants).
Updated docs and release note text to reflect the new wire keys.
Updated unit-test comments/assertion expectations that reference the renamed keys.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
release-notes.d/unreleased/1937.md	Updates release note wording to the new `kv_transfer_params.remote_kv_source` key.
pkg/sidecar/proxy/proxy.go	Renames the P2P-related `kv_transfer_params` field constants to the new vLLM key names.
pkg/sidecar/proxy/connector_p2p.go	Updates inline documentation around the injected `kv_transfer_params` block names.
pkg/sidecar/proxy/connector_p2p_test.go	Updates test expectations/comments to reference `remote_decoder` / `remote_prefiller`.
docs/disaggregation.md	Updates disaggregation documentation examples to use the renamed keys.

nilig · 2026-07-23T15:53:49Z

+// p2pSourceParams builds the kv_transfer_params.remote_kv_source block for a
+// pull from sourceHostPort's OffloadingConnector P2P tier. The kv_request_id is its
 // own fresh UUID: in P2P mode it is consumer-side only.


The quoted comment was reworded in 9cac78f, and 43aea48 renames the requestFieldP2P* constants to match the wire keys. Remaining p2p references name the secondary-tier type, which is unchanged in vLLM.

The kv_transfer_params nesting key is remote_kv_source; comments and test descriptions that still call it the p2p key or p2p params read as if the secondary-tier type (p2p, unchanged) were the request-field key. Signed-off-by: nilig <nili.ifergan@gmail.com>

roytman · 2026-07-23T15:31:32Z

 }

-// addP2PPullToPrefill adds the OffloadingConnector p2p pull block to a prefill
+// addP2PPullToPrefill adds the OffloadingConnector P2P pull block to a prefill


There is some inconsistency in the names and comments P2P vs remote_kv_source, as in the comment https://github.com/llm-d/llm-d-router/pull/2140/changes#r3639185910

Renamed the constants after their wire keys in 43aea48: requestFieldRemoteDecoder, requestFieldRemotePrefiller, requestFieldRemoteKVSource, matching the sibling request-field constants, and swept the test comments that still said decode/prefill/p2p. P2P now only names the mechanism: the OffloadingConnector tier (type "p2p", unchanged in vLLM) and its protocol functions.

The requestFieldP2P* constants held remote_* wire keys, so the names no longer described the values. Naming them for their values matches the sibling request-field constants and leaves P2P to name only the OffloadingConnector tier and its protocol functions. Also sweeps test comments and one spec title that still named the sub-dicts by the old decode/prefill/p2p keys. Signed-off-by: nilig <nili.ifergan@gmail.com>

roytman

/lgtm

nilig requested review from a team, roytman and shmuelk as code owners July 22, 2026 13:09

nilig requested review from ahg-g and elevran July 22, 2026 13:09

github-actions Bot added kind/bug Categorizes issue or PR as related to a bug. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 22, 2026

nilig mentioned this pull request Jul 22, 2026

P2P KV cache sharing support in the router #1929

Open

5 tasks

shmuelk reviewed Jul 22, 2026

View reviewed changes

docs: wrap the renamed disaggregation paragraphs at 80 columns

8b389d4

Review feedback on llm-d#2140: the two protocol paragraphs were single long lines; wrapped for reviewable diffs. No content change. Signed-off-by: nilig <nili.ifergan@gmail.com>

github-actions Bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jul 23, 2026

roytman requested a review from Copilot July 23, 2026 15:04

Copilot started reviewing on behalf of roytman July 23, 2026 15:04 View session

Copilot AI reviewed Jul 23, 2026

View reviewed changes

roytman reviewed Jul 23, 2026

View reviewed changes

github-actions Bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 23, 2026

roytman approved these changes Jul 24, 2026

View reviewed changes

vMaroon approved these changes Jul 24, 2026

View reviewed changes

vMaroon merged commit b25dd66 into llm-d:main Jul 24, 2026
29 checks passed

llm-d-router-release-notes Bot mentioned this pull request Jul 24, 2026

docs: add release-note fragment for PR #2140 #2177

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM#2140

fix(sidecar): rename P2P kv_transfer_params sub-dict keys to match vLLM#2140
vMaroon merged 4 commits into
llm-d:mainfrom
nilig:fix/p2p-kv-transfer-param-rename

nilig commented Jul 22, 2026

Uh oh!

shmuelk Jul 22, 2026

Uh oh!

nilig Jul 23, 2026

Uh oh!

shmuelk Jul 22, 2026

Uh oh!

nilig Jul 23, 2026

Uh oh!

shmuelk commented Jul 22, 2026

Uh oh!

nilig commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

nilig Jul 23, 2026

Uh oh!

roytman Jul 23, 2026

Uh oh!

nilig Jul 23, 2026

Uh oh!

roytman left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Conversation

nilig commented Jul 22, 2026

Uh oh!

shmuelk Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

nilig Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

shmuelk Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

nilig Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

shmuelk commented Jul 22, 2026

Uh oh!

nilig commented Jul 23, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

nilig Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

roytman Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

nilig Jul 23, 2026

Choose a reason for hiding this comment

Uh oh!

roytman left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants