MultiKueue: Support priority-based / ordered cluster selection

[mpsanj]

Hi Team,

Support for priority/weight-based cluster selection in MultiKueue, so that workloads
are preferentially dispatched to higher-priority worker clusters before falling back
to lower-priority ones.

Why is this needed?

Currently, MultiKueue uses a race model — workloads are dispatched to all worker
clusters listed in MultiKueueConfig.spec.clusters simultaneously, and the first
cluster to admit wins. There is no mechanism to express cluster preference or ordering.

Use case

We run 3 worker clusters across regions (e.g., JP, US, SG) for GPU workloads. We want
to prefer scheduling on our primary cluster (cheaper/closer), and only overflow to
secondary and tertiary clusters when the primary lacks resources.

Proposed solution

Add a priority or weight field to MultiKueueConfig.spec.clusters:

apiVersion: kueue.x-k8s.io/v1beta2
kind: MultiKueueConfig
metadata:
  name: multikueue-default
spec:
  clusters:
    - name: "jp-osa"
      priority: 1        # highest priority, try first
    - name: "us-002"
      priority: 2        # try if jp-osa is full
    - name: "sg-001"
      priority: 3        # last resort

Expected behavior with ordered dispatch

1. Submit workload to the highest-priority cluster only
2. If that cluster's ClusterQueue cannot admit (quota full / pending) within a configurable timeout, try the next cluster
3. Clean up remote workloads on clusters that lost
4. Fall back to race-all if all clusters are above threshold

Environment

• Kueue version: v0.16.1
• Kubernetes version: 1.33
• MultiKueue with 3 worker clusters

If there are already workaround, please suggest.

Thank You,
Sanjay

[mimowo]

Currently, MultiKueue uses a race model — workloads are dispatched to all worker

Actually, the incremental mode doesn't do it, but it goes in steps by three clusters, ptal https://github.com/kubernetes-sigs/kueue/blob/main/apis/config/v1beta2/configuration_types.go#L301-L303

We run 3 worker clusters across regions (e.g., JP, US, SG) for GPU workloads. We want
to prefer scheduling on our primary cluster (cheaper/closer), and only overflow to
secondary and tertiary clusters when the primary lacks resources.

Would the incremental mode be sufficient for your use-case if there was some configuration to allow going in steps one-by-one?

[olekzabl]

/area multikueue

[tenzen-y]

We run 3 worker clusters across regions (e.g., JP, US, SG) for GPU workloads. We want
to prefer scheduling on our primary cluster (cheaper/closer), and only overflow to
secondary and tertiary clusters when the primary lacks resources.

Would the incremental mode be sufficient for your use-case if there was some configuration to allow going in steps one-by-one?

I think so too. Additionally, if you want to handle cluster selection fine-grained, you can implement your custom multikueue-dispatcher.

[mpsanj]

Currently, MultiKueue uses a race model — workloads are dispatched to all worker

Actually, the incremental mode doesn't do it, but it goes in steps by three clusters, ptal https://github.com/kubernetes-sigs/kueue/blob/main/apis/config/v1beta2/configuration_types.go#L301-L303

We run 3 worker clusters across regions (e.g., JP, US, SG) for GPU workloads. We want
to prefer scheduling on our primary cluster (cheaper/closer), and only overflow to
secondary and tertiary clusters when the primary lacks resources.

Would the incremental mode be sufficient for your use-case if there was some configuration to allow going in steps one-by-one?

Yes, incremental mode with a configurable step size would solve our use case. With 3 clusters and step=1, the dispatcher would try the first cluster, and only expand to the next if it can't admit.

Thanks for the quick response!

[mimowo]

Awesome! would you have cycles yourself to work on the PR? We need to align on the API field name (maybe IncrementalModeConfig.StepSize), so a small KEP update for MultiKueue would be great.

Since this is a small change, I think we could probably couple KEP and impl in one PR.

[tenzen-y]

Awesome! would you have cycles yourself to work on the PR? We need to align on the API field name (maybe IncrementalModeConfig.StepSize), so a small KEP update for MultiKueue would be great.

Since this is a small change, I think we could probably couple KEP and impl in one PR.

I remember that we discussed exposing the step size to user facing place.
So, basically, +1 on this enhancement.