What would you like to be added:
/feature
A "hub mode" that generalizes the EPP scheduling candidate one tier up, from pod to destination (a local InferencePool, a peer cluster, or an external provider), so the existing Filter/Score/Pick pipeline handles multi-cluster and multi-provider routing. Destinations are Endpoints in the hub's own InferencePool and the cross-cluster signal is the spoke EPP's existing pool-aggregate metrics, so there is no new scheduler and no core scheduling changes; the additions are a candidate producer and a locality scorer.
Design Doc: Hub and Spoke: multi-cluster llm-d
Why is this needed:
llm-d has no upstream path for multi-cluster and external-provider routing today. The EPP already owns the scheduling decision and the load signal within a cluster, so generalizing its candidate to a destination extends that to multi-cluster on the same pipeline, with no new component.
related work items
What would you like to be added:
/feature
A "hub mode" that generalizes the EPP scheduling candidate one tier up, from pod to destination (a local InferencePool, a peer cluster, or an external provider), so the existing Filter/Score/Pick pipeline handles multi-cluster and multi-provider routing. Destinations are
Endpoints in the hub's own InferencePool and the cross-cluster signal is the spoke EPP's existing pool-aggregate metrics, so there is no new scheduler and no core scheduling changes; the additions are a candidate producer and a locality scorer.Design Doc: Hub and Spoke: multi-cluster llm-d
Why is this needed:
llm-d has no upstream path for multi-cluster and external-provider routing today. The EPP already owns the scheduling decision and the load signal within a cluster, so generalizing its candidate to a destination extends that to multi-cluster on the same pipeline, with no new component.
related work items
file-discoveryto support cluster endpoints #1903 file-discovery: cluster (hostname) endpoints (PR feat(datalayer): extend file-discovery to support cluster endpoints #1927), incl. EPP-side hostname->IP resolution