A guide for using Kueue as a TAS scheduler for LWS

[ahg-g]

What would you like to be added:

A user guide that shows how to deploy Kueue as a topology-aware scheduler for LWS.

Ideally we come up with a helm chart in Kueue that allows deploying Kueue as a TAS scheduler only, abstracting the concepts of ClusterQueue, Flavor and LocalQueue; It would take a couple of parameters, the topology labels and the accelerator type.

We may need to automate creating LocalQueues, perhaps we should have Kueue create a default LocalQueue in each namespace.

[Edwinhr716]

Current state of Kueue + LWS https://kueue.sigs.k8s.io/docs/tasks/run/leaderworkerset/

[Edwinhr716]

Ideally we come up with a helm chart in Kueue that allows deploying Kueue as a TAS scheduler only, abstracting the concepts of ClusterQueue, Flavor and LocalQueue; It would take a couple of parameters, the topology labels and the accelerator type.

We may need to automate creating LocalQueues, perhaps we should have Kueue create a default LocalQueue in each namespace.

Should we open an issue on the Kueue side as well for this ask?

[ahg-g]

I think it is good to start the discussion on this, yes

[kannon92]

The other relevant documentation would be https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/

[kannon92]

hmm.

cc @mimowo @tenzen-y

We can discuss here or in Kueue but this is an interesting feature request..

[kannon92]

I think this is a valid use case for LWS but in general it seems there is a lot of interest in collocating Training and Serving workloads.

In your request, it sounds like you want to only serve a single tenant designed for LWS (maybe llm-d) and you are not really focused on multi tenant?

[kannon92]

We may need to automate creating LocalQueues, perhaps we should have Kueue create a default LocalQueue in each namespace.

Starting in 0.12, Kueue has a localQueueDefaulting feature where it will use the "default" localQueue if it exists.

I'm not entirely sure how you automate that with a helm chart as this has to be created for each namespace you support.

[ahg-g]

Not via helm, I want Kueue to create the default localQueue if it doesn't exist in each namespace that points to a default CQ.

[kannon92]

Not via helm, I want Kueue to create the default localQueue if it doesn't exist in each namespace that points to a default CQ.

That is a feature request. Right now Kueue requires one to create a local queue named default to use LocalQueueDefaulting. I don’t think Kueue has the ability to create local queues.

[ahg-g]

yes, it is a feature request; but what does this have to do with creating namespaces? the ask is to create a default localQueue in existing namespaces in the cluster.

[mimowo]

Sounds like a useful feature request for Kueue, and I'm open to the contribution. Some questions I have, but don't seem blockers:

1. what is "each namespace" (maybe we can use the configuration's NamespaceSelector for that)
2. what is "default CQ" - maybe that could be provided as a Helm template parameter (? )
3. what if new namespaces are added after Kueue's installation

Also, as we add more logic into Helm we may need to consider automated e2e tests for that (currently we have none).

cc @mwielgus @mwysokin

[kannon92]

yes, it is a feature request; but what does this have to do with creating namespaces? the ask is to create a default localQueue in existing namespaces in the cluster.

Sorry that was a typo.

[ahg-g]

Sounds like a useful feature request for Kueue, and I'm open to the contribution. Some questions I have, but don't seem blockers:

1. what is "each namespace" (maybe we can use the configuration's NamespaceSelector for that)

Every namespace in the cluster, existing and new; we could restrict that using a Kueue component config parameter that selects in which namespaces the LocalQueue should be created.

2. what is "default CQ" - maybe that could be provided as a Helm template parameter (? )

Yes, this part is easy, a parameter in the helm chart that is also passed to the Kueue controller which, as I suggest later, will set it in each default LocalQueue it creates

3. what if new namespaces are added after Kueue's installation>
   Also, as we add more logic into Helm we may need to consider automated e2e tests for that (currently we have none).

The ask is not for the helm chart to create the default LocalQueue, but for the Kueue controller to do that for every namespace.

cc @mwielgus @mwysokin