What happened?
When two or more pods share a single physical GPU through one DRA ResourceClaim
(the claim's status.reservedFor lists more than one consumer — a normal
time-slicing / MPS setup), the scheduler counts that one physical device once
per consuming pod instead of once per device.
On a node whose shared device pushes the counted GPU usage above the node's
physical GPU count, the node's idle GPU count goes negative and the node becomes
unschedulable for every task — including pods that request no GPU at all. The
rejection surfaces as a misleading message, e.g.:
<node>: Node didn't have enough resources: GPUs, requested: 0, used: 2, capacity: 1
no nodes with enough resources were found: 1 node(s) didn't have enough resources: GPUs.
Concretely, on a single-GPU node with one claim shared by two pods, the scheduler
reports used: 2, capacity: 1, idle -1. A 0-GPU pod pinned to that node stays
Pending with the message above.
Root cause is in the node resource accounting: addTaskResources adds each
task's DRA GPU count from its allocated ResourceClaimInfo to UsedVector, and
nothing dedups a device that is already contributed by another pod on the same
node sharing the same claim. (pkg/scheduler/api/node_info/node_info.go →
addTaskResources; DRA GPU count comes from
pkg/common/resources/dra.go → ExtractDRAGPUResourcesFromClaims, which counts
per requested device with no cross-pod awareness.)
What did you expect to happen?
A physical GPU shared by several pods through one ResourceClaim should count
towards the node's used GPUs once, so idle never goes below zero and the node
stays schedulable for other workloads.
How can we reproduce it?
- On a single-GPU node, create one
ResourceClaim requesting that GPU with a
sharing strategy that allows multiple consumers (DRA time-slicing or MPS).
- Schedule two pods that both reference the claim, so
status.reservedFor holds
both. Verify via scheduler logs (schedulingshard spec.args.v: "7") that the
node shows used: 2, capacity: 1, idle GPU -1.
- Schedule any third pod (even one requesting 0 GPUs) with a nodeSelector onto
that node — it stays Pending with "didn't have enough resources: GPUs".
Environment
- Kubernetes version: 1.36
- KAI Scheduler version: v0.16.4
- DRA: NVIDIA DRA driver (
gpu.nvidia.com), device-plugin disabled
(devicePlugin.enabled=false); scheduling driven entirely by DRA
- Hardware: NVIDIA RTX PRO GPUs, one physical GPU shared by two pods via MPS
What happened?
When two or more pods share a single physical GPU through one DRA
ResourceClaim(the claim's
status.reservedForlists more than one consumer — a normaltime-slicing / MPS setup), the scheduler counts that one physical device once
per consuming pod instead of once per device.
On a node whose shared device pushes the counted GPU usage above the node's
physical GPU count, the node's idle GPU count goes negative and the node becomes
unschedulable for every task — including pods that request no GPU at all. The
rejection surfaces as a misleading message, e.g.:
Concretely, on a single-GPU node with one claim shared by two pods, the scheduler
reports
used: 2, capacity: 1, idle-1. A 0-GPU pod pinned to that node staysPendingwith the message above.Root cause is in the node resource accounting:
addTaskResourcesadds eachtask's DRA GPU count from its allocated
ResourceClaimInfotoUsedVector, andnothing dedups a device that is already contributed by another pod on the same
node sharing the same claim. (
pkg/scheduler/api/node_info/node_info.go→addTaskResources; DRA GPU count comes frompkg/common/resources/dra.go→ExtractDRAGPUResourcesFromClaims, which countsper requested device with no cross-pod awareness.)
What did you expect to happen?
A physical GPU shared by several pods through one ResourceClaim should count
towards the node's used GPUs once, so idle never goes below zero and the node
stays schedulable for other workloads.
How can we reproduce it?
ResourceClaimrequesting that GPU with asharing strategy that allows multiple consumers (DRA time-slicing or MPS).
status.reservedForholdsboth. Verify via scheduler logs (
schedulingshardspec.args.v: "7") that thenode shows
used: 2, capacity: 1, idle GPU-1.that node — it stays
Pendingwith "didn't have enough resources: GPUs".Environment
gpu.nvidia.com), device-plugin disabled(
devicePlugin.enabled=false); scheduling driven entirely by DRA