[RFC] Support kubernetes gang scheduling as a pod group provider

[kannon92]

What would you like to be added:

gang scheduling in LWS is an interesting feature.

There is an upstream effort in k8s around gang scheduling: kubernetes/enhancements#5558

It would be great if we could provide a support via alpha APIs once the PR merges.

This way we could support gang scheduling intree.

Why is this needed:
See above
Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[kannon92]

ref: kubernetes-sigs/jobset#969

For Jobset we are hoping to leverage k8s gang scheduling feature.

[Edwinhr716]

/assign

I'm already looking into this, hope to have either a KEP or a high level design doc soon to go over what it will look like from LWS side. We will have to wait until Gang-Scheduling is in alpha for implementation though right?

[ardaguclu]

We will have to wait until Gang-Scheduling is in alpha for implementation though right?

Gang-Scheduling will be released as alpha behind a feature gate. In my opinion, it would be ideal integrating it when it is promoted to at least beta. Yet, I understand that people eagerly wait this functionality in LWS in this face paced environment. So that we need to have some smart mechanism to distinguish Gang-Scheduling is enabled in the cluster or not.

[ahg-g]

I think we should integrate in alpha and have a feature gate in LWS (so it is a deployment time knob).

The alpha feature in kube will need feedback to get to beta, and so we should try and provide that since LWS and the uses cases it supports are one of the reasons for which gang scheduling is being built in upstream k8s.

[Edwinhr716]

I think we should integrate in alpha and have a feature gate in LWS

Why have a specific feature gate on LWS? Can't we do something similar with what we do with the MaxUnavailable feature where we mention that you need to enable the upstream flag?

[ahg-g]

I don't think so, LWS will need to create and watch on the Workload resource, and that will only work only on clusters with the Workload API defined

[ardaguclu]

The alpha feature in kube will need feedback to get to beta, and so we should try and provide that since LWS and the uses cases it supports are one of the reasons for which gang scheduling is being built in upstream k8s.

I agree with you and it would be nice to support this. But there are many edge cases, since, as you mentioned, APIs need to be watched by LWS.

• LWS is running on cluster, feature gate is enabled in cluster afterwards.
• LWS hasn't been installed yet, feature gate is enabled in cluster
• LWS is running with this feature gate, but feature gate is disabled in the cluster afterwards-- although this is rare, but still a possibility that should be handled gracefully in LWS.

[ahg-g]

The above also applies in beta, consider an LWS deployed on a cluster with a version that doesn't support the feature and then it gets upgraded to a version that does

[Edwinhr716]

Wrote this doc to get the conversation started https://docs.google.com/document/d/1QlcIBtR2KyOKYRUTGubhhxuy7NfjHs1fXMJlvdUCyhM/edit?tab=t.0

[JesseStutler]

I have a question, there's no doubt that we need to support upstream gang scheduling. Do we still need to evolve custom schedulers #619 and #620?

[ahg-g]

I would recommend against it. My expectation is that custom schedulers will align with the upstream workload api