Skip to content

fix(router): exclude cached KV transfers from prefill compute accounting #15812

Open

Description

@NVShreyas

Problem

The KV router uses transfer-cost-discounted cache overlap both for worker selection and for active-prefill load accounting. With CPU or disk KV offloading, cached tokens can therefore be counted as GPU prefill compute. This inflates the selected worker's load and can reduce prefix affinity.

Scope

Account for the selected worker/DP rank's actual cached blocks across device, host, and disk tiers when computing active-prefill load. Keep transfer-cost weighting in worker selection. Classify requests waiting for remote KV as queued decode work rather than queued prefill where that correction is not already present.

No dedicated remote-KV backlog telemetry, new configuration, overload threshold, or automatic instrumented-scheduler activation is needed for this correction.

Implementation: #15209. The same compute-only correction is carried on the MiniMax release build branch in #14856.

Activity

  1. added
    dynamo-runtimeRelates to the dynamo-runtime component
    routerRelates to routing, KV-aware routing, etc.
    language::rustIssues/PRs that reference Rust code
    on Oct 7, 2026
  2. added a commit that references this issue on Oct 7, 2026
    1ab1d1d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dynamo-runtimeRelates to the dynamo-runtime componentlanguage::rustIssues/PRs that reference Rust coderouterRelates to routing, KV-aware routing, etc.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions