fix(nemotron_3.5_super): pin a vllm-router build that also fixes pref… - #3080
Open
sdevare-nv wants to merge 1 commit into
Open
fix(nemotron_3.5_super): pin a vllm-router build that also fixes pref…#3080sdevare-nv wants to merge 1 commit into
sdevare-nv wants to merge 1 commit into
Conversation
…ill affinity The wheel build pins the head of vllm-project/router#216, which fixes the decode-node hot spot (issue #197). Shipping #216 alone turns out to be worse than not shipping it for prefill-heavy benchmarks such as SWE-bench Pro. #216 makes the worker load counters honest, and that switches on a second latent bug: cache_aware decides whether to use prefix affinity from the fleet-wide load spread, so one hot worker discards affinity for every request -- including requests whose own worker is idle. Under P/D disaggregation that gate is open almost permanently, because prefill worker load counts queued requests as well as running ones. Routing degenerates to shortest-queue, already-cached prompts get recomputed, prefill saturates and decode starves behind it. Repin to the head of vllm-project/router#238, which carries #216 unchanged plus a fix applying the same load check per request, against the worker the request actually wants, rather than against the fleet maximum. Still fetched from vllm-project/router by bare SHA, exactly as the #216 pin already was. Signed-off-by: Sugam Devare <sdevare@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Repins the
vllm-routerbuild used by the Super 3.5 eval container.build_vllm_router_wheel.shcurrently pins the head ofvllm-project/router#216, which
fixes the decode-node hot spot (issue #197). Shipping #216 on its own turns out to
be worse than not shipping it at all for prefill-heavy benchmarks like SWE-bench
Pro.
#216 makes the worker load counters honest, and that switches on a second latent
bug.
cache_awaredecides whether to use prefix affinity from the fleet-wideload spread, so a single hot worker discards affinity for every request —
including requests whose own worker is idle. Under P/D disaggregation that gate is
open almost permanently, because prefill worker load counts queued requests as well
as running ones. Routing degenerates to shortest-queue, prompts that were already
cached get recomputed, prefill saturates, and decode starves behind it.
The pin now points at a branch carrying #216 plus a fix that applies the same
load check per request — against the worker the request actually wants, rather than
against the fleet maximum:
vllm-project/router#238.
Both pins are plain commit SHAs fetched from
vllm-project/router— GitHub servesany commit reachable from a ref, and a PR head is a ref even when the branch lives on
a contributor's fork. No fork remote is involved. Repin to a released commit once
these land upstream.
Also in the wheel build:
VLLM_ROUTER_SRCbuilds from a local router checkout instead of the pinned SHA,for iterating on a fork. Mounted read-only and copied into the build root, so the
build never writes into the source tree.
cargo check --all-targets. Neither the wheel build norcargo test --libcompiles the
vllm-routerbinary target, so a break insrc/main.rsshippedsilently before.
cache_awarepolicy tests run alongside the existing load-accounting ones.Heads-up for reviewers
This pins an unmerged PR head, exactly as the existing #216 pin already does. Happy
to hold until #238 merges upstream if you would rather not carry an unmerged pin.
Testing
Validated on SWE-bench Pro (4 prefill / 6 decode) against two baselines on the same
checkpoint and vLLM config: the pre-#216 router, and #216 alone. With this pin the
prefill prefix-cache hit rate and end-to-end throughput return to pre-#216
behaviour while keeping #216's decode hot-spot fix.