[Feature] [Alpha] Application Priority and Preemption

[RainbowMango]

Summary
Currently, the karmada-scheduler operates based on a First-In-First-Out scheduling order. In some real-world scenarios, like AI training, some jobs are more critical than others and require preferential treatment in terms of scheduling and resource allocation.

We need the feature to enable users to specify the importance of an application. Applications with higher priority are scheduled ahead of other applications with lower priority. When member clusters do not have enough capacity for running a high priority application, the scheduler preempts and removes lower priority applications in order to make room for the high priority application.

Proposal:

• Application Priority and Preemption

Author:

• @LeonZh0u
• @seanlaii
• @wengyao04
• @whitewindmills
• @zclyne (Yifan Zhang)

References:

• [Scheduler] about priority scheduling #4938
• Proposal in Google Doc: https://docs.google.com/document/d/1MixmgLwnmiRrukyFP25JvE2hqiqB3f_JKT12tXiOsbc/edit#

Iteration Tasks -- Part-1: Scheduling based on application priority:

• API change: Introduce priority class to PropagationPolicy (@seanlaii, Add priorityClassName in PropagationPolicy & ClusterPropagationPolicy #5962)
• API change: Introduce priority to ResourceBinding (@seanlaii, Add Priority and PreemptionPolicy fields in ResourceBinding #5963)
• Feature gate: Introduce a feature gate for this feature. (@LeonZh0u)
• Controller: Build application priority when creating ResourceBinding (@LeonZh0u )
• webhook: Introduce validation to restrict PriorityClass modification
• scheduler: Introduce PriorityQueue to replace the current RateLimitedQueue. (@whitewindmills)
• E2E tests
• Documentation

Iteration Tasks -- Part-2: Preemption in case of member clusters lack of capacity:

• API change: Introduce preemptionPolicy to ResourceBinding
• TBD

Welcome Join us
If anyone is interested in this feature or would like to participate in its development, please leave a comment in this issue to claim the task you are interested in.

Note that this feature is still under discussion, and the listed iteration tasks might be subject to change.

[seanlaii]

Hi @RainbowMango , I can take the first two tasks first. Thank you so much for this!
Also, for the second task, we might need to add preemptionPolicy in the resourceBinding as well.

[RainbowMango]

Sure thanks. Assigned and updated.
By the way, do you know Yifan's GitHub handle? He is the co-author of this proposal, right?

[whitewindmills]

@RainbowMango
in the part-I, we don't need any feature gates.

[whitewindmills]

I have already implemented a priority queue demo, so I will continue to work on it. please assign it to me.
/assign
/assign @seanlaii

[RainbowMango]

@RainbowMango in the part-I, we don't need any feature gates.

Not sure about it yet. We probably need a feature gate to isolate the changes. Let's keep it on the list for now.

[seanlaii]

Sure thanks. Assigned and updated.

By the way, do you know Yifan's GitHub handle? He is the co-author of this proposal, right?

Yes, this is his github: https://github.com/zclyne. Thank you!

[zhzhuang-zju]

webhook: Introduce validation to restrict PriorityClass modification

I'm quite interested in this task. If no one else claims it, I can be assigned it.