What happened:
Several users reported the infinite preemption loop in fairsharing over the last months.
The root cause of the issue is the following. Preempting targets from same-CQ as the incoming workload causes its DRS to decrease, which makes more targets available. But after gathering all available targets, we proceed to fillBackWorkloads, where some same-CQ targets are removed, which increases DRS again which may make some other targets illegal (from the fair sharing perspective). Its possible to construct an example in which such a behavior leads to the infinite loop 7e9aaf4.
I was able to only reproduce it when both fairsharing strategies were in use. But, its possible that a simpler reproduction may be possible.
In the integration test I reproduced the behavior which we observed in the logs of the issue in the real cluster, where after a preemption, a preemptee gets admitted (instead of the preemptor).
In the unit test I showed an example in which two workloads may preempt each other (which is shown to be impossible in https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/). The reason the proof does not cover this case is that one of the preemptions that happen in this case is illegal (preemptor has higher DRS).
What you expected to happen:
No infinite preemption.
How to reproduce it (as minimally and precisely as possible):
7e9aaf4
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version):
- Kueue version (use
git describe --tags --dirty --always):
- Cloud provider or hardware configuration:
- OS (e.g:
cat /etc/os-release):
- Kernel (e.g.
uname -a):
- Install tools:
- Others:
What happened:
Several users reported the infinite preemption loop in fairsharing over the last months.
The root cause of the issue is the following. Preempting targets from same-CQ as the incoming workload causes its DRS to decrease, which makes more targets available. But after gathering all available targets, we proceed to
fillBackWorkloads, where some same-CQ targets are removed, which increases DRS again which may make some other targets illegal (from the fair sharing perspective). Its possible to construct an example in which such a behavior leads to the infinite loop 7e9aaf4.I was able to only reproduce it when both fairsharing strategies were in use. But, its possible that a simpler reproduction may be possible.
In the integration test I reproduced the behavior which we observed in the logs of the issue in the real cluster, where after a preemption, a preemptee gets admitted (instead of the preemptor).
In the unit test I showed an example in which two workloads may preempt each other (which is shown to be impossible in https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/). The reason the proof does not cover this case is that one of the preemptions that happen in this case is illegal (preemptor has higher DRS).
What you expected to happen:
No infinite preemption.
How to reproduce it (as minimally and precisely as possible):
7e9aaf4
Anything else we need to know?:
Environment:
kubectl version):git describe --tags --dirty --always):cat /etc/os-release):uname -a):