fix(scheduler): count shared DRA GPU device once per node

A single physical GPU shared by several pods through one ResourceClaim
(status.reservedFor with more than one entry) was counted once per
consuming pod in the node used vector. On a node whose shared device
pushed the used GPU count above physical capacity, IdleVector went
negative and the node became unschedulable for every task, including
GPU-less ones, surfacing as a misleading "didn't have enough resources:
GPUs" message.

Track a per-device reference count on the node (keyed by
driver/pool/device) so a shared physical device contributes to the used
GPU count exactly once, with symmetric add/remove accounting.

Signed-off-by: Raya Solano <raya@mbinf.de>

Author: Raya Solano

CHANGELOG.md

@@ -4,6 +4,44 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [Unreleased]
+
+### Fixed
+- Count a physical GPU shared by several pods through one DRA ResourceClaim (`status.reservedFor` with more than one entry) once per node instead of once per consuming pod. Per-task accounting added the device for every consumer, so on a node whose shared device pushed the used GPU count above physical capacity, `IdleVector` went negative and the node became unschedulable for every task — including GPU-less ones — surfacing as a misleading "didn't have enough resources: GPUs" message. [#1930](https://github.com/kai-scheduler/KAI-Scheduler/issues/1930)
+
+## [v0.16.4] - 2026-07-12
+
+### Added
+- Publish FIPS-enabled image variants (`<version>-fips`) for every release, built with the Go toolchain's native FIPS 140-3 mode (`GOFIPS140`), and added a `global.fips` Helm value (default `false`) that appends `-fips` to every resolved image tag ([guide](docs/fips/README.md)). [#1867](https://github.com/kai-scheduler/KAI-Scheduler/issues/1867)
+- Added `global.nodePoolLabelKey` Helm value to configure `spec.global.nodePoolLabelKey` in the Config CR for KAI sharding [#1774](https://github.com/kai-scheduler/KAI-Scheduler/issues/1774).
+
+### Fixed
+- Scoped the operator's informer cache for Pods, Leases and EndpointSlices to the KAI namespace and stripped managed fields from cached objects. Since v0.15.0 the operator cached every such object in the cluster, so its memory grew with cluster size and exceeded the default 256Mi limit on large clusters. [#1780](https://github.com/kai-scheduler/KAI-Scheduler/issues/1780)
+- Block NaN value for fraction in the pod admission [#1798](https://github.com/kai-scheduler/KAI-Scheduler/issues/1798) [davidLif](https://github.com/davidLif)
+- In the fractional admission checks, check that the fractional value can be parsed as a quantity. [#1798](https://github.com/kai-scheduler/KAI-Scheduler/issues/1798) [davidLif](https://github.com/davidLif)
+
+## [v0.16.2] - 2026-06-30
+
+### Fixed
+- Fixed scheduler pod cache memory growth by transforming cached Pods to retain only scheduler-relevant container fields while stripping large literal env values and managed fields [#1646](https://github.com/kai-scheduler/KAI-Scheduler/issues/1646)
+- Fix the MinNodeGPUMemoryMiB calculation in the scheduler. This affected allocations for fractional pod requesting gpu "gpu-memory". [#1795](https://github.com/kai-scheduler/KAI-Scheduler/issues/1795) [davidLif](https://github.com/davidLif)
+- Use the maximum gpu size ine the cluster rather then the minimum when checking a potential overLimit or isNonPreemptebleOverquota for a pod. [#1792](https://github.com/kai-scheduler/KAI-Scheduler/issues/1795) [davidLif](https://github.com/davidLif)
+
+## [v0.16.1] - 2026-06-28
+
+### Added
+- Added `global.resourceReservation.createNamespace` Helm value (default `true`) to allow disabling creation of the resource-reservation namespace, for embedding KAI in a parent chart that creates the namespace itself.
+- Added `global.resourceReservation.createServiceAccount` Helm value (default `true`) to allow disabling creation of the resource-reservation ServiceAccount, for embedding KAI in a parent chart that creates the ServiceAccount itself.
+- Added `defaultPriorityClasses.enabled` Helm value (default `true`) for installations that manage KAI PriorityClasses externally.
+
+### Changed
+- Podgrouper now preserves an existing PodGroup's topology constraint when the workload does not specify one, so an externally-assigned topology is not overwritten. Workload topology annotations still take precedence when present.
+
+### Fixed
+- Restricted Helm post-delete cleanup to KAI operator-managed Deployments and preserved externally managed `kai-config` resources when `kaiConfigDeployer.enabled=false`.
+- Scheduler cache now filters terminal Pods at watch time to reduce memory use, while still watching Pods bound by other schedulers so their resource usage is counted in allocatable calculations. [#1645](https://github.com/kai-scheduler/KAI-Scheduler/issues/1645) [enoodle](https://github.com/enoodle)
+- Fixed reclaim abandoning valid over-quota victims when an unrelated under-deserved queue appeared earlier in victim ordering. [#1750](https://github.com/kai-scheduler/KAI-Scheduler/issues/1750)
+
 ## [v0.16.0] - 2026-06-24
 
 ### Added

pkg/common/constants/constants.go

@@ -9,6 +9,7 @@ const (
 	AppLabelName              = "app"
 	NvidiaGpuResource         = "nvidia.com/gpu"
 	NvidiaGpuMemory           = "nvidia.com/gpu.memory"
+	NvidiaGpuDraDriver        = "gpu.nvidia.com"
 	NvidiaMigResourcePrefix   = "nvidia.com/mig-"
 	GpuResource               = "gpu"
 	UnlimitedResourceQuantity = float64(-1)

pkg/scheduler/api/node_info/dra_shared_device_info.go

@@ -0,0 +1,100 @@
+// Copyright 2025 NVIDIA CORPORATION
+// SPDX-License-Identifier: Apache-2.0
+
+package node_info
+
+import (
+	resourceapi "k8s.io/api/resource/v1"
+
+	commonconstants "github.com/kai-scheduler/KAI-scheduler/pkg/common/constants"
+	"github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/pod_info"
+	"github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/resource_info"
+)
+
+// A single physical DRA device may be shared by several pods through one
+// ResourceClaim with more than one entry in status.reservedFor. Each such pod
+// carries the same allocated device in its ResourceClaimInfo, so counting the
+// device once per pod (the naive per-task accounting) inflates the node's used
+// GPU count above physical capacity and drives IdleVector negative, making the
+// whole node unschedulable. draSharedDeviceRefCount tracks how many pods on the
+// node currently reference each allocated device (keyed by driver/pool/device)
+// so a shared device contributes to UsedVector exactly once.
+
+// draDeviceKey uniquely identifies a physical DRA device on the node.
+func draDeviceKey(result resourceapi.DeviceRequestAllocationResult) string {
+	return result.Driver + "/" + result.Pool + "/" + result.Device
+}
+
+// allocatedGPUDeviceKeys returns the keys of all GPU devices allocated to the
+// task via DRA ResourceClaims. Non-GPU devices are ignored: they are not part
+// of the GPU accounting that this dedup protects.
+func (ni *NodeInfo) allocatedGPUDeviceKeys(task *pod_info.PodInfo) []string {
+	var keys []string
+	for _, claimAllocation := range task.ResourceClaimInfo {
+		if claimAllocation == nil || claimAllocation.Allocation == nil {
+			continue
+		}
+		for _, result := range claimAllocation.Allocation.Devices.Results {
+			if !isGPUDRADriver(result.Driver) {
+				continue
+			}
+			keys = append(keys, draDeviceKey(result))
+		}
+	}
+	return keys
+}
+
+// isGPUDRADriver reports whether the DRA driver name belongs to an NVIDIA GPU
+// device. The device-class name is not present on the allocation result, so the
+// driver name is used instead.
+func isGPUDRADriver(driver string) bool {
+	return driver == commonconstants.NvidiaGpuDraDriver
+}
+
+// dedupSharedDRAGpus removes from resourcesToTrack the GPU count that would
+// double-count physical DRA devices already referenced by other pods on the
+// node. It also updates the node's per-device reference count. It must be
+// called once per addTaskResources, before the vector is added to UsedVector.
+func (ni *NodeInfo) dedupSharedDRAGpus(task *pod_info.PodInfo, resourcesToTrack resource_info.ResourceVector) {
+	alreadyCounted := 0.0
+	for _, key := range ni.allocatedGPUDeviceKeys(task) {
+		if ni.DRASharedDeviceRefCount[key] > 0 {
+			// Another pod on this node already contributed this physical
+			// device to the used vector: do not count it again.
+			alreadyCounted++
+		}
+		ni.DRASharedDeviceRefCount[key]++
+	}
+
+	if alreadyCounted > 0 {
+		current := resourcesToTrack.Get(resource_info.GPUIndex)
+		resourcesToTrack.Set(resource_info.GPUIndex, current-alreadyCounted)
+	}
+}
+
+// releaseSharedDRAGpus is the inverse of dedupSharedDRAGpus: it decrements the
+// per-device reference count and adds back the GPU count for devices that
+// remain referenced by other pods (and were therefore never subtracted on this
+// task's removal path). It must be called once per removeTaskResources.
+func (ni *NodeInfo) releaseSharedDRAGpus(task *pod_info.PodInfo, resourcesToTrack resource_info.ResourceVector) {
+	stillShared := 0.0
+	for _, key := range ni.allocatedGPUDeviceKeys(task) {
+		if ni.DRASharedDeviceRefCount[key] > 1 {
+			// The device stays referenced by another pod after this removal:
+			// it must remain in the used vector, so this task's removal must
+			// not subtract it.
+			stillShared++
+		}
+		if ni.DRASharedDeviceRefCount[key] > 0 {
+			ni.DRASharedDeviceRefCount[key]--
+		}
+		if ni.DRASharedDeviceRefCount[key] == 0 {
+			delete(ni.DRASharedDeviceRefCount, key)
+		}
+	}
+
+	if stillShared > 0 {
+		current := resourcesToTrack.Get(resource_info.GPUIndex)
+		resourcesToTrack.Set(resource_info.GPUIndex, current-stillShared)
+	}
+}

pkg/scheduler/api/node_info/dra_shared_device_info_test.go

@@ -0,0 +1,171 @@
+// Copyright 2025 NVIDIA CORPORATION
+// SPDX-License-Identifier: Apache-2.0
+
+package node_info
+
+import (
+	"testing"
+
+	"github.com/stretchr/testify/assert"
+	"go.uber.org/mock/gomock"
+	v1 "k8s.io/api/core/v1"
+	resourceapi "k8s.io/api/resource/v1"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
+	"k8s.io/utils/ptr"
+
+	commonconstants "github.com/kai-scheduler/KAI-scheduler/pkg/common/constants"
+	"github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/common_info"
+	"github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/pod_affinity"
+	"github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/pod_info"
+	"github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/resource_info"
+)
+
+const gpuDeviceClass = "gpu.nvidia.com"
+
+// sharedGPUClaim builds a ResourceClaim that requests one GPU device and is
+// allocated to the given physical device. The same claim object is referenced
+// by every pod that shares the device (status.reservedFor with multiple
+// entries), matching a DRA time-slicing / MPS setup.
+func sharedGPUClaim(name, namespace, driver, pool, device string) *resourceapi.ResourceClaim {
+	return &resourceapi.ResourceClaim{
+		ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: namespace},
+		Spec: resourceapi.ResourceClaimSpec{
+			Devices: resourceapi.DeviceClaim{
+				Requests: []resourceapi.DeviceRequest{
+					{
+						Name: "gpu",
+						Exactly: &resourceapi.ExactDeviceRequest{
+							DeviceClassName: gpuDeviceClass,
+							AllocationMode:  resourceapi.DeviceAllocationModeExactCount,
+							Count:           1,
+						},
+					},
+				},
+			},
+		},
+		Status: resourceapi.ResourceClaimStatus{
+			Allocation: &resourceapi.AllocationResult{
+				Devices: resourceapi.DeviceAllocationResult{
+					Results: []resourceapi.DeviceRequestAllocationResult{
+						{Request: "gpu", Driver: driver, Pool: pool, Device: device},
+					},
+				},
+			},
+		},
+	}
+}
+
+// draConsumerPod builds a running pod that consumes the given claim by name.
+func draConsumerPod(name, namespace, nodeName, claimName string) *v1.Pod {
+	pod := common_info.BuildPod(namespace, name, nodeName, v1.PodRunning,
+		common_info.BuildResourceList("1000m", "1G"), []metav1.OwnerReference{},
+		map[string]string{}, map[string]string{
+			pod_info.ReceivedResourceTypeAnnotationName: string(pod_info.ReceivedTypeRegular),
+			commonconstants.PodGroupAnnotationForPod:    common_info.FakePogGroupId,
+		})
+	pod.Spec.ResourceClaims = []v1.PodResourceClaim{
+		{Name: claimName, ResourceClaimName: ptr.To(claimName)},
+	}
+	return pod
+}
+
+// newGPUNodeInfo builds a NodeInfo for a node with the given whole-GPU count,
+// wired with a mock pod-affinity that expects addPods AddPod and rmPods
+// RemovePod calls, and a vectorMap that knows the DRA GPU device class.
+func newGPUNodeInfo(t *testing.T, name, gpuCount string, addPods, rmPods int) (*NodeInfo, *resource_info.ResourceVectorMap) {
+	node := common_info.BuildNode(name, common_info.BuildResourceListWithGPU("8000m", "16G", gpuCount))
+
+	ctrl := gomock.NewController(t)
+	affinity := pod_affinity.NewMockNodePodAffinityInfo(ctrl)
+	affinity.EXPECT().AddPod(gomock.Any()).Times(addPods)
+	affinity.EXPECT().RemovePod(gomock.Any()).Times(rmPods)
+
+	vectorMap := resource_info.NewResourceVectorMap()
+	for resourceName := range node.Status.Allocatable {
+		vectorMap.AddResource(resourceName)
+	}
+	// DRA GPU counts are tracked under the device-class resource name.
+	vectorMap.AddResource(v1.ResourceName(gpuDeviceClass))
+
+	return NewNodeInfo(node, affinity, vectorMap), vectorMap
+}
+
+// TestAddTask_SharedDRAClaimCountedOnce is the regression test for the
+// shared-ResourceClaim double count. Two pods share one physical GPU through a
+// single claim (reservedFor has both). Naive per-task accounting adds the GPU
+// once per pod, driving the node's used GPU count to 2 on a 1-GPU node and
+// IdleVector negative, which makes the node unschedulable for every task. The
+// fix must keep the used count at exactly 1.
+//
+// This exercises the real accounting path (AddTask -> addTaskResources ->
+// UsedVector), not the dedup helper in isolation, so it fails if the dedup is
+// not wired into AddTask.
+func TestAddTask_SharedDRAClaimCountedOnce(t *testing.T) {
+	ni, vectorMap := newGPUNodeInfo(t, "atlas", "1", 2, 0)
+
+	claim := sharedGPUClaim("voice-atlas-shared", "voice-pipeline", gpuDeviceClass, "atlas", "gpu-0")
+	pod1 := draConsumerPod("voice-tts", "voice-pipeline", "atlas", "voice-atlas-shared")
+	pod2 := draConsumerPod("voice-worker", "voice-pipeline", "atlas", "voice-atlas-shared")
+
+	task1 := pod_info.NewTaskInfo(pod1, vectorMap, pod_info.TaskInfoOptions{
+		DraPodClaims: []*resourceapi.ResourceClaim{claim},
+	})
+	task2 := pod_info.NewTaskInfo(pod2, vectorMap, pod_info.TaskInfoOptions{
+		DraPodClaims: []*resourceapi.ResourceClaim{claim},
+	})
+
+	assert.NoError(t, ni.AddTask(task1))
+	assert.Equal(t, 1.0, ni.UsedVector.Get(resource_info.GPUIndex),
+		"first consumer of the shared device must be counted")
+
+	assert.NoError(t, ni.AddTask(task2))
+	assert.Equal(t, 1.0, ni.UsedVector.Get(resource_info.GPUIndex),
+		"second consumer of the same shared device must not be double-counted")
+
+	idleGPUs, _ := ni.GetSumOfIdleGPUs()
+	assert.Equal(t, 0.0, idleGPUs, "idle GPUs must be 0, never negative, on a fully-shared 1-GPU node")
+}
+
+// TestAddRemoveTask_SharedDRAClaimSymmetry verifies the used count follows the
+// number of distinct physical devices as consumers are added and removed: it
+// stays at 1 while any consumer of the shared device remains, and returns to 0
+// only when the last one leaves.
+func TestAddRemoveTask_SharedDRAClaimSymmetry(t *testing.T) {
+	ni, vectorMap := newGPUNodeInfo(t, "atlas", "1", 2, 2)
+
+	claim := sharedGPUClaim("voice-atlas-shared", "voice-pipeline", gpuDeviceClass, "atlas", "gpu-0")
+	pod1 := draConsumerPod("voice-tts", "voice-pipeline", "atlas", "voice-atlas-shared")
+	pod2 := draConsumerPod("voice-worker", "voice-pipeline", "atlas", "voice-atlas-shared")
+	task1 := pod_info.NewTaskInfo(pod1, vectorMap, pod_info.TaskInfoOptions{DraPodClaims: []*resourceapi.ResourceClaim{claim}})
+	task2 := pod_info.NewTaskInfo(pod2, vectorMap, pod_info.TaskInfoOptions{DraPodClaims: []*resourceapi.ResourceClaim{claim}})
+
+	assert.NoError(t, ni.AddTask(task1))
+	assert.NoError(t, ni.AddTask(task2))
+	assert.Equal(t, 1.0, ni.UsedVector.Get(resource_info.GPUIndex))
+
+	assert.NoError(t, ni.RemoveTask(task2))
+	assert.Equal(t, 1.0, ni.UsedVector.Get(resource_info.GPUIndex),
+		"removing one of two consumers must keep the shared device counted")
+
+	assert.NoError(t, ni.RemoveTask(task1))
+	assert.Equal(t, 0.0, ni.UsedVector.Get(resource_info.GPUIndex),
+		"removing the last consumer must release the shared device")
+}
+
+// TestAddTask_DistinctDRADevicesEachCounted guards against over-dedup: two pods
+// on two different physical GPUs (own claim each) must both be counted.
+func TestAddTask_DistinctDRADevicesEachCounted(t *testing.T) {
+	ni, vectorMap := newGPUNodeInfo(t, "nyx", "2", 2, 0)
+
+	claimA := sharedGPUClaim("coder-claim", "vllm", gpuDeviceClass, "nyx", "gpu-4")
+	claimB := sharedGPUClaim("gemma-claim", "vllm", gpuDeviceClass, "nyx", "gpu-6")
+	podA := draConsumerPod("coder", "vllm", "nyx", "coder-claim")
+	podB := draConsumerPod("gemma", "vllm", "nyx", "gemma-claim")
+	taskA := pod_info.NewTaskInfo(podA, vectorMap, pod_info.TaskInfoOptions{DraPodClaims: []*resourceapi.ResourceClaim{claimA}})
+	taskB := pod_info.NewTaskInfo(podB, vectorMap, pod_info.TaskInfoOptions{DraPodClaims: []*resourceapi.ResourceClaim{claimB}})
+
+	assert.NoError(t, ni.AddTask(taskA))
+	assert.NoError(t, ni.AddTask(taskB))
+	assert.Equal(t, 2.0, ni.UsedVector.Get(resource_info.GPUIndex),
+		"two distinct physical devices must both be counted")
+}

pkg/scheduler/api/node_info/gpu_sharing_node_info.go

@@ -21,6 +21,12 @@ type GpuSharingNodeInfo struct {
 	UsedSharedGPUsMemory      map[string]int64
 	ReleasingSharedGPUsMemory map[string]int64
 	AllocatedSharedGPUsMemory map[string]int64
+
+	// DRASharedDeviceRefCount counts how many pods on the node reference each
+	// physical DRA GPU device (keyed by driver/pool/device). A device shared
+	// by several pods through one ResourceClaim (status.reservedFor with more
+	// than one entry) must contribute to the node's used GPU count only once.
+	DRASharedDeviceRefCount map[string]int
 }
 
 func newGpuSharingNodeInfo() *GpuSharingNodeInfo {
@@ -30,6 +36,8 @@ func newGpuSharingNodeInfo() *GpuSharingNodeInfo {
 		UsedSharedGPUsMemory:      make(map[string]int64),
 		ReleasingSharedGPUsMemory: make(map[string]int64),
 		AllocatedSharedGPUsMemory: make(map[string]int64),
+
+		DRASharedDeviceRefCount: make(map[string]int),
 	}
 }
 
@@ -48,6 +56,9 @@ func (g *GpuSharingNodeInfo) Clone() *GpuSharingNodeInfo {
 	for k, v := range g.AllocatedSharedGPUsMemory {
 		gpuSharingNodeInfo.AllocatedSharedGPUsMemory[k] = v
 	}
+	for k, v := range g.DRASharedDeviceRefCount {
+		gpuSharingNodeInfo.DRASharedDeviceRefCount[k] = v
+	}
 
 	return gpuSharingNodeInfo
 }

pkg/scheduler/api/node_info/node_info.go

@@ -466,6 +466,10 @@ func (ni *NodeInfo) addTaskResources(task *pod_info.PodInfo) {
 		resourcesToTrackVector.Set(resource_info.GPUIndex, 0)
 	}
 
+	// A physical DRA device shared by several pods (one ResourceClaim with
+	// multiple reservedFor entries) must be counted once, not once per pod.
+	ni.dedupSharedDRAGpus(task, resourcesToTrackVector)
+
 	ni.UsedVector.Add(resourcesToTrackVector)
 
 	switch task.Status {
@@ -516,6 +520,10 @@ func (ni *NodeInfo) removeTaskResources(task *pod_info.PodInfo) {
 		resourcesToTrackVector.Set(resource_info.GPUIndex, 0)
 	}
 
+	// Mirror of dedupSharedDRAGpus: keep a shared physical DRA device in the
+	// used vector as long as another pod on the node still references it.
+	ni.releaseSharedDRAGpus(task, resourcesToTrackVector)
+
 	ni.UsedVector.Sub(resourcesToTrackVector)
 
 	switch task.Status {