[MultiKueue] The eviction is not working #7350
Description
What happened:
I'm trying to create high priority job to preempt low priority job but it is not working as expect. The workload stuck on.
Status from high priority job:
- lastTransitionTime: "2025-10-23T06:33:18Z"
message: 'couldn''t assign flavors to pod set main: insufficient unused quota
for memory in flavor default, 1953125Ki more needed. Pending the preemption
of 1 workload(s)'
observedGeneration: 1
reason: Pending
status: "False"
type: QuotaReservedStatus from low priority job:
conditions:
- lastTransitionTime: "2025-10-23T06:33:17Z"
message: Quota reserved in ClusterQueue q1
observedGeneration: 1
reason: QuotaReserved
status: "True"
type: QuotaReserved
- lastTransitionTime: "2025-10-23T06:33:18Z"
message: 'Preempted to accommodate a workload (UID: 9f542104-a9e2-4344-ad2e-9c136204da18,
JobUID: a20d0a6a-64a3-46ce-9b95-8cfb3bf6e675) due to prioritization in the
ClusterQueue'
observedGeneration: 1
reason: Preempted
status: "True"
type: Evicted
- lastTransitionTime: "2025-10-23T06:33:18Z"
message: The workload is admitted
observedGeneration: 1
reason: Admitted
status: "True"
type: Admitted
- lastTransitionTime: "2025-10-23T06:33:18Z"
message: 'Preempted to accommodate a workload (UID: 9f542104-a9e2-4344-ad2e-9c136204da18,
JobUID: a20d0a6a-64a3-46ce-9b95-8cfb3bf6e675) due to prioritization in the
ClusterQueue'
reason: InClusterQueue
status: "True"
type: PreemptedLogs from the manager cluster occurring in a loop:
2025-10-23T07:13:10.668141483Z DEBUG events recorder/recorder.go:104 couldn't assign flavors to pod set main: insufficient unused quota for memory in flavor default, 1953125Ki more needed. Pending the preemption of 1 workload(s) {"type": "Warning", "object": {"kind":"Workload","namespace":"multikueue-vggw9","name":"job-job2-f620a","uid":"4cc488da-107c-4814-8248-b44ac9462816","apiVersion":"kueue.x-k8s.io/v1beta1","resourceVersion":"14153"}, "reason": "Pending"}
2025-10-23T07:13:10.76844517Z LEVEL(-3) scheduler queue/manager.go:661 Obtained ClusterQueue heads {"schedulingCycle": 12168, "count": 1}
2025-10-23T07:13:10.768592836Z LEVEL(-3) scheduler scheduler/fair_sharing_iterator.go:69 Returning workload from ClusterQueue without Cohort {"schedulingCycle": 12168, "clusterQueue": {"name":"q1"}, "workload": {"name":"job-job2-f620a","namespace":"multikueue-vggw9"}}
2025-10-23T07:13:10.768610128Z LEVEL(-2) scheduler scheduler/scheduler.go:252 Attempting to schedule workload {"schedulingCycle": 12168, "workload": {"name":"job-job2-f620a","namespace":"multikueue-vggw9"}, "clusterQueue": {"name":"q1"}}
2025-10-23T07:13:10.768633711Z LEVEL(-3) scheduler preemption/preemption.go:190 Preemption ongoing {"schedulingCycle": 12168, "workload": {"name":"job-job2-f620a","namespace":"multikueue-vggw9"}, "clusterQueue": {"name":"q1"}, "targetWorkload": {"name":"job-job1-6f320","namespace":"multikueue-vggw9"}, "preemptingWorkload": {"name":"job-job2-f620a","namespace":"multikueue-vggw9"}}
2025-10-23T07:13:10.768676628Z LEVEL(-3) scheduler scheduler/logging.go:42 Workload evaluated for admission {"schedulingCycle": 12168, "workload": {"name":"job-job2-f620a","namespace":"multikueue-vggw9"}, "clusterQueue": {"name":"q1"}, "status": "", "reason": "couldn't assign flavors to pod set main: insufficient unused quota for memory in flavor default, 1953125Ki more needed. Pending the preemption of 1 workload(s)"}
2025-10-23T07:13:10.768709211Z LEVEL(-2) scheduler scheduler/scheduler.go:778 Workload re-queued {"schedulingCycle": 12168, "workload": {"name":"job-job2-f620a","namespace":"multikueue-vggw9"}, "clusterQueue": {"name":"q1"}, "queue": {"name":"q1","namespace":"multikueue-vggw9"}, "requeueReason": "PendingPreemption", "added": true, "status": ""}
What you expected to happen:
The high priority job should be successfully admitted.
How to reproduce it (as minimally and precisely as possible):
- Create low priority job
- Create high priority job
Anything else we need to know?:
Environment:
- Kubernetes version (use
kubectl version): - Kueue version (use
git describe --tags --dirty --always): - Cloud provider or hardware configuration:
- OS (e.g:
cat /etc/os-release): - Kernel (e.g.
uname -a): - Install tools:
- Others: