[Bugfix] Expose LMCache controller reply/heartbeat ports on the router Service - #1029
[Bugfix] Expose LMCache controller reply/heartbeat ports on the router Service#1029bazakeliad wants to merge 2 commits into
Conversation
…r Service
The chart tells the engines to reach the controller on a Service port the chart
never opens, so KV-aware routing fails silently.
Three templates disagree today:
- deployment-router.yaml launches the router with --lmcache-controller-reply-port
and --lmcache-controller-heartbeat-port from routerSpec, so the router listens
on them, but declares only a hardcoded containerPort 9000.
- deployment-vllm-multi.yaml sets
LMCACHE_CONTROLLER_REPLY_URL={release}-router-service:{controllerReplyPort},
so every engine is told to connect there.
- service-router.yaml exposes 9000 only.
Registration then blocks inside ZMQ with no error on either side: no exception,
no log line, and lookups simply return nothing, so kvaware degenerates to QPS
routing while appearing healthy.
Also fixes the pull port itself: containerPort and the Service port were both
hardcoded to 9000 and ignored routerSpec.lmcacheControllerPort, so any non-default
value pointed the Service at a port the router was not listening on.
lmcacheControllerReplyPort and lmcacheControllerHeartbeatPort were already read by
deployment-router.yaml but undocumented in values.yaml; both are now documented and
present in values.schema.json.
Ports are rendered only when the corresponding value is set, so charts that do not
configure the controller are unaffected: helm template with default values produces
byte-identical output before and after this change.
Signed-off-by: Eliad Bazak <bazakeliad@gmail.com>
There was a problem hiding this comment.
Code Review
This pull request introduces configuration options for the LMCache controller reply and heartbeat ports in the Helm chart, enabling worker registration and re-registration. The reviewer suggests adding fail-fast template validations to ensure these ports are provided when routingLogic is set to kvaware, and updating the documentation in values.yaml and values.schema.json to clearly reflect these requirements.
Review feedback: make it explicit in values.yaml, and mirror it in values.schema.json, that these ports matter when routingLogic is kvaware. The reply port is documented as required - registration blocks silently without it, which is the bug this PR fixes. The heartbeat port is documented as strongly recommended rather than required, because that is what it is: kvaware works without it until the router restarts, at which point workers cannot re-register and routing degrades to QPS until the engines restart too. Signed-off-by: Eliad Bazak <bazakeliad@gmail.com>
|
Thanks for the review. I've taken the documentation suggestions and pushed them in 24981d5 — One wording change from the suggestion: I documented the heartbeat port as strongly recommended rather than required, because that is what we measured. Without it, On the fail-fast validationsI'd rather not add them in this PR, for two reasons: 1. It would break existing deployments. No one sets these ports today — that is precisely the bug this PR fixes, since the chart never exposed them. A 2. For the heartbeat port the premise doesn't hold. It genuinely is optional, per the measurement above, so a hard requirement would be incorrect. That said, I think the underlying instinct is right, and it is the same one behind this PR: a silent degradation is worse than a loud failure. Turning that into a validation is a breaking change and deserves to be the maintainers' decision rather than a rider on a bugfix. Happy to send it as a follow-up PR if you'd like it — possibly as a warning path, or gated behind a chart-major bump. Current state: defaults render byte-identically, |
Description
The chart points the engines at a router Service port that the chart never opens, so KV-aware
routing fails silently.
Three templates disagree on
maintoday:helm/templates/deployment-router.yaml--lmcache-controller-reply-port/--lmcache-controller-heartbeat-portfromrouterSpec.*— so the router listens on them — but declares only a hardcodedcontainerPort: 9000helm/templates/deployment-vllm-multi.yamlLMCACHE_CONTROLLER_REPLY_URL={{ .Release.Name }}-router-service:{{ controllerReplyPort }}— so every engine is told to connect therehelm/templates/service-router.yaml9000onlyWorker registration then blocks inside ZMQ with no error on either side: no exception, no log
line, and
lookup()simply returns nothing — sokvawaredegenerates to QPS routing while thedeployment looks healthy. It took us a while to find, because every surface reports success.
This also fixes the pull port itself: both the
containerPortand the Service port werehardcoded to
9000and ignoredrouterSpec.lmcacheControllerPort, so any non-default valuepointed the Service at a port the router was not listening on.
Finally,
lmcacheControllerReplyPortandlmcacheControllerHeartbeatPortwere already read bydeployment-router.yamlbut were undocumented invalues.yaml. Both are now documented, andvalues.schema.jsonis updated to match.Backward compatibility
Each port renders only when its value is set, so charts that do not configure the controller
are untouched. Verified:
With the ports configured:
the router Service now renders:
with matching named
containerPorts on the router Deployment. (lmcache-hbeatrather thanlmcache-heartbeatbecause Kubernetes port names are limited to 15 characters.)How this was found
Deploying the KV-aware setup on a 2×A10 OpenShift cluster. The symptom was that
kvawarerouting appeared to work but never showed cache affinity. We worked around it with
oc patch svc ... --type=jsonadding the two ports, which confirmed the diagnosis, and aresending the proper fix upstream.
helm lintpasses;values.schema.jsonregenerated to matchvalues.yaml.-swhen doinggit commit[Bugfix],[Feat], and[CI].