Skip to content

feat(epp): hub mode for multi-cluster routing #1907

Description

@hexfusion

What would you like to be added:

/feature

A "hub mode" that generalizes the EPP scheduling candidate one tier up, from pod to destination (a local InferencePool, a peer cluster, or an external provider), so the existing Filter/Score/Pick pipeline handles multi-cluster and multi-provider routing. Destinations are Endpoints in the hub's own InferencePool and the cross-cluster signal is the spoke EPP's existing pool-aggregate metrics, so there is no new scheduler and no core scheduling changes; the additions are a candidate producer and a locality scorer.

Design Doc: Hub and Spoke: multi-cluster llm-d

Why is this needed:

llm-d has no upstream path for multi-cluster and external-provider routing today. The EPP already owns the scheduling decision and the load signal within a cluster, so generalizing its candidate to a destination extends that to multi-cluster on the same pipeline, with no new component.

Image

related work items

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions