You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(router): exclude cached KV transfers from prefill compute accounting #15812
The KV router uses transfer-cost-discounted cache overlap both for worker selection and for active-prefill load accounting. With CPU or disk KV offloading, cached tokens can therefore be counted as GPU prefill compute. This inflates the selected worker's load and can reduce prefix affinity.
Scope
Account for the selected worker/DP rank's actual cached blocks across device, host, and disk tiers when computing active-prefill load. Keep transfer-cost weighting in worker selection. Classify requests waiting for remote KV as queued decode work rather than queued prefill where that correction is not already present.
No dedicated remote-KV backlog telemetry, new configuration, overload threshold, or automatic instrumented-scheduler activation is needed for this correction.
Implementation: #15209. The same compute-only correction is carried on the MiniMax release build branch in #14856.
Activity
fix(router): limit PR to compute-only prefill accounting