Add PDB for kai scheduler

Signed-off-by: dttung2905 <ttdao.2015@accountancy.smu.edu.sg>

Author: dttung2905

CHANGELOG.md

@@ -4,6 +4,40 @@ All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
+## [Unreleased]
+
+### Added
+- Added `global.resourceReservation.createNamespace` Helm value (default `true`) to allow disabling creation of the resource-reservation namespace, for embedding KAI in a parent chart that creates the namespace itself.
+- Added `global.resourceReservation.createServiceAccount` Helm value (default `true`) to allow disabling creation of the resource-reservation ServiceAccount, for embedding KAI in a parent chart that creates the ServiceAccount itself.
+- Added `defaultPriorityClasses.enabled` Helm value (default `true`) for installations that manage KAI PriorityClasses externally.
+- Added GitOps/ArgoCD install support ([guide](docs/gitops/README.md)): `kaiConfig.render` Helm value (default `false`) renders the `kai-config` Config CR inline as a tracked release resource (mutually exclusive with `kaiConfigDeployer.enabled`), `openshift` value (default `false`) forces OpenShift mode where `lookup` auto-detection is unavailable under offline rendering, and ArgoCD `PostDelete` hook and `Prune=false` annotations (requires ArgoCD >= 2.10). [#1794](https://github.com/kai-scheduler/KAI-Scheduler/issues/1794) [#1751](https://github.com/kai-scheduler/KAI-Scheduler/issues/1751)
+- Added **topology level aliases**: a `Topology` level may declare an `alias` (e.g. `rack`), usable in place of the raw node label key in a workload's `requiredTopologyLevel`/`preferredTopologyLevel`. Aliases are one-to-one (unique within the Topology and must not collide with a `nodeLabel`, enforced by a new Topology validating webhook) and may be edited freely (the `levels` immutability rule now freezes only the `nodeLabel` structure). When a level has no alias, behavior is unchanged and raw label keys keep working. [#1498](https://github.com/kai-scheduler/KAI-Scheduler/issues/1498)
+- Added support for configuring admission Pod Disruption Budget via Helm values (`admission.podDisruptionBudget`) [#1490](https://github.com/kai-scheduler/KAI-Scheduler/pull/1490) [dttung2905](https://github.com/dttung2905)
+- Added support for configuring scheduler Pod Disruption Budget via Helm values (`scheduler.podDisruptionBudget`) when running multiple replicas per scheduling shard
+
+### Changed
+- Scenario search now skips re-simulating equivalent victim-set candidates that already failed simulation for the same pending job (reclaim, preempt, consolidation). Skipped candidates are recorded as `state="duplicate"` in `scenario_search_scenarios_total`. [#1719](https://github.com/kai-scheduler/KAI-Scheduler/issues/1719)
+- Removed unused `queuecontroller.certSecretName` and `admission.certSecretName` Helm values; webhook TLS secrets are created and managed by the operator (`queue-webhook-tls-secret`, `kai-admission-webhook-tls-secret`). [#1791](https://github.com/kai-scheduler/KAI-Scheduler/pull/1791) [dttung2905](https://github.com/dttung2905)
+- Podgrouper now preserves an existing PodGroup's topology constraint when the workload does not specify one, so an externally-assigned topology is not overwritten. Workload topology annotations still take precedence when present.
+
+### Fixed
+- Scenario search no longer leaks a rejected scenario's victim nodes into the probe's feasible-node set: the solver now rolls back feasible-node additions after validator-rejected and errored simulations, not only after cleanly unsolved ones. [#1719](https://github.com/kai-scheduler/KAI-Scheduler/issues/1719)
+- Scoped the operator's informer cache for Pods, Leases and EndpointSlices to the KAI namespace and stripped managed fields from cached objects. Since v0.15.0 the operator cached every such object in the cluster, so its memory grew with cluster size and exceeded the default 256Mi limit on large clusters. [#1780](https://github.com/kai-scheduler/KAI-Scheduler/issues/1780)
+- Reduced transient scheduler allocations during large reclaim operations by comparing proportion queue state and cached resource vectors directly instead of repeatedly materializing resource maps.
+- Scheduler now exits on 401 Unauthorized API responses instead of retrying indefinitely with a stale ServiceAccount token. [#1817](https://github.com/kai-scheduler/KAI-Scheduler/issues/1817)
+- Reduced scheduler memory use during large reclaim operations by removing redundant per-job-pair min-runtime protection caching; effective min-runtime durations remain cached per queue pair. [#1808](https://github.com/kai-scheduler/KAI-Scheduler/issues/1808)
+- Fixed reclaim abandoning valid over-quota victims when an unrelated under-deserved queue appeared earlier in victim ordering. [#1750](https://github.com/kai-scheduler/KAI-Scheduler/issues/1750)
+- Restricted Helm post-delete cleanup to KAI operator-managed Deployments and preserved externally managed `kai-config` resources when `kaiConfigDeployer.enabled=false`.
+- Scheduler cache now filters terminal Pods at watch time to reduce memory use, while still watching Pods bound by other schedulers so their resource usage is counted in allocatable calculations. [#1645](https://github.com/kai-scheduler/KAI-Scheduler/issues/1645) [enoodle](https://github.com/enoodle)
+- Fix the MinNodeGPUMemoryMiB calculation in the scheduler. This affected allocations for fractional pod requesting gpu "gpu-memory". [#1792](https://github.com/kai-scheduler/KAI-Scheduler/issues/1792) [davidLif](https://github.com/davidLif)
+- Use the maximum gpu size ine the cluster rather then the minimum when checking a potential overLimit or isNonPreemptebleOverquota for a pod. [#1792](https://github.com/kai-scheduler/KAI-Scheduler/issues/1792) [davidLif](https://github.com/davidLif)
+- Reduced allocation churn in the scheduler hot path: cached `Schedulable()` result as a package-level singleton and lazily formatted `logNodeSetsPluginResult` node names only when verbose logging is enabled.
+- Block NaN value for fraction in the pod admission [#1798](https://github.com/kai-scheduler/KAI-Scheduler/issues/1798) [davidLif](https://github.com/davidLif)
+- In the fractional admission checks, check that the fractional value can be parsed as a quantity. [#1798](https://github.com/kai-scheduler/KAI-Scheduler/issues/1798) [davidLif](https://github.com/davidLif)
+- Podgrouper now rejects negative PyTorch replica indexes and LWS worker indexes, and caps the number of subgroups created for block-level segmentation at 10000 to avoid unbounded PodGroup fan-out. [davidLif](https://github.com/davidLif)
+- Fixed GPU-sharing pods with dotted pod names generating invalid ConfigMap-backed volume names. Volume names are now sanitized to valid DNS labels while preserving original ConfigMap references used for shared-GPU injection.
+
+
 ## [v0.16.0] - 2026-06-24
 
 ### Added

deployments/kai-scheduler/templates/_helpers.tpl

@@ -283,6 +283,13 @@ spec:
       affinity:
         {{- toYaml .Values.scheduler.affinity | nindent 8 }}
       {{- end }}
+      podDisruptionBudget:
+        {{- if hasKey .Values.scheduler.podDisruptionBudget "enabled" }}
+        enabled: {{ .Values.scheduler.podDisruptionBudget.enabled }}
+        {{- end }}
+        {{- if hasKey .Values.scheduler.podDisruptionBudget "maxUnavailable" }}
+        maxUnavailable: {{ .Values.scheduler.podDisruptionBudget.maxUnavailable }}
+        {{- end }}
     {{- if and .Values.scheduler.ports .Values.scheduler.ports.metricsPort }}
     schedulerService:
       port: {{ .Values.scheduler.ports.metricsPort }}

deployments/kai-scheduler/tests/scheduler_pdb_test.yaml

@@ -0,0 +1,35 @@
+# Copyright 2025 NVIDIA CORPORATION
+# SPDX-License-Identifier: Apache-2.0
+
+suite: scheduler podDisruptionBudget in kai-config
+
+set:
+  kaiConfigDeployer:
+    enabled: false
+  kaiConfig:
+    render: true
+
+templates:
+  - kai-config.yaml
+tests:
+  - it: renders scheduler podDisruptionBudget from values
+    set:
+      scheduler.podDisruptionBudget:
+        enabled: true
+        maxUnavailable: 1
+    asserts:
+      - equal:
+          path: spec.scheduler.service.podDisruptionBudget.enabled
+          value: true
+      - equal:
+          path: spec.scheduler.service.podDisruptionBudget.maxUnavailable
+          value: 1
+
+  - it: renders disabled PDB without maxUnavailable when only enabled is set in test values
+    set:
+      scheduler.podDisruptionBudget:
+        enabled: false
+    asserts:
+      - equal:
+          path: spec.scheduler.service.podDisruptionBudget.enabled
+          value: false

deployments/kai-scheduler/values.yaml

@@ -163,6 +163,11 @@ scheduler:
   actions: {}
   # Alpha/experimental scenario search time budgets. Values are Go duration strings.
   scenarioSearchBudgets: {}
+  # PDB is created only when scheduler replicas > 1 (global.replicaCount / scheduler.replicas).
+  # For HA: set operator.replicaCount: 2 (or override scheduler.replicas in Config).
+  podDisruptionBudget:
+    enabled: true
+    maxUnavailable: 1
 
 # defaultShard controls the chart-managed "default" SchedulingShard CR,
 # whose spec is populated from the scheduler.* values above.

pkg/operator/operands/common/common.go

@@ -36,6 +36,7 @@ const (
 // PodDisruptionBudgetImplementedServices lists operand resource names with operator-side PDB creation.
 var PodDisruptionBudgetImplementedServices = map[string]struct{}{
 	"admission": {},
+	"scheduler": {},
 }
 
 func PodDisruptionBudgetImplemented(serviceName string) bool {

pkg/operator/operands/common/common_test.go

@@ -538,9 +538,9 @@ var _ = Describe("PodDisruptionBudgetForKAIConfig", func() {
 
 var _ = Describe("PodDisruptionBudgetImplementedServices", func() {
 	It("only lists operands with operator-side PDB creation", func() {
-		Expect(PodDisruptionBudgetImplementedServices).To(HaveLen(1))
+		Expect(PodDisruptionBudgetImplementedServices).To(HaveLen(2))
 		Expect(PodDisruptionBudgetImplemented("admission")).To(BeTrue())
+		Expect(PodDisruptionBudgetImplemented("scheduler")).To(BeTrue())
 		Expect(PodDisruptionBudgetImplemented("binder")).To(BeFalse())
-		Expect(PodDisruptionBudgetImplemented("scheduler")).To(BeFalse())
 	})
 })

pkg/operator/operands/scheduler/resources_for_shard.go

@@ -375,6 +375,21 @@ func DeploymentName(config *kaiv1.Config, shard *kaiv1.SchedulingShard) string {
 	return fmt.Sprintf("%s-%s", *config.Spec.Global.SchedulerName, shard.Name)
 }
 
+func (s *SchedulerForShard) podDisruptionBudgetForShard(
+	ctx context.Context, readerClient client.Reader,
+	kaiConfig *kaiv1.Config, shard *kaiv1.SchedulingShard,
+) (client.Object, error) {
+	config := kaiConfig.Spec.Scheduler
+	return common.PodDisruptionBudgetForKAIConfig(
+		ctx,
+		readerClient,
+		kaiConfig.Spec.Namespace,
+		DeploymentName(kaiConfig, shard),
+		config.Replicas,
+		config.Service,
+	)
+}
+
 // serviceName for the per-shard scheduler Service.
 func serviceName(config *kaiv1.Config, shard *kaiv1.SchedulingShard) string {
 	return fmt.Sprintf("%s-%s", *config.Spec.Global.SchedulerName, shard.Name)

pkg/operator/operands/scheduler/resources_test.go

@@ -15,6 +15,7 @@ import (
 
 	"github.com/kai-scheduler/KAI-scheduler/cmd/scheduler/app/options"
 	kaiv1 "github.com/kai-scheduler/KAI-scheduler/pkg/apis/kai/v1"
+	"github.com/kai-scheduler/KAI-scheduler/pkg/apis/kai/v1/common"
 	kaiprometheus "github.com/kai-scheduler/KAI-scheduler/pkg/apis/kai/v1/prometheus"
 	kaiv1qc "github.com/kai-scheduler/KAI-scheduler/pkg/apis/kai/v1/queue_controller"
 	kaiv1scheduler "github.com/kai-scheduler/KAI-scheduler/pkg/apis/kai/v1/scheduler"
@@ -27,6 +28,7 @@ import (
 	"github.com/stretchr/testify/require"
 	appsv1 "k8s.io/api/apps/v1"
 	corev1 "k8s.io/api/core/v1"
+	policyv1 "k8s.io/api/policy/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 	"k8s.io/utils/ptr"
 	"sigs.k8s.io/controller-runtime/pkg/client/fake"
@@ -1413,3 +1415,81 @@ func TestGetUsageDBConfig(t *testing.T) {
 		})
 	}
 }
+
+func TestPodDisruptionBudgetForShard(t *testing.T) {
+	ctx := context.Background()
+	client := fake.NewClientBuilder().Build()
+
+	shard := &kaiv1.SchedulingShard{
+		ObjectMeta: metav1.ObjectMeta{Name: "default"},
+	}
+	shard.Spec.SetDefaultsWhereNeeded()
+
+	tests := []struct {
+		name              string
+		replicas          int32
+		pdbEnabled        bool
+		maxUnavailable    int32
+		expectPDBCreation bool
+	}{
+		{
+			name:              "skip PDB when replicas is one",
+			replicas:          1,
+			pdbEnabled:        true,
+			maxUnavailable:    1,
+			expectPDBCreation: false,
+		},
+		{
+			name:              "create PDB when replicas greater than one and enabled",
+			replicas:          2,
+			pdbEnabled:        true,
+			maxUnavailable:    1,
+			expectPDBCreation: true,
+		},
+		{
+			name:              "skip PDB when disabled",
+			replicas:          3,
+			pdbEnabled:        false,
+			maxUnavailable:    1,
+			expectPDBCreation: false,
+		},
+		{
+			name:              "custom maxUnavailable",
+			replicas:          2,
+			pdbEnabled:        true,
+			maxUnavailable:    2,
+			expectPDBCreation: true,
+		},
+	}
+
+	for _, tt := range tests {
+		t.Run(tt.name, func(t *testing.T) {
+			config := &kaiv1.Config{}
+			config.Spec.SetDefaultsWhereNeeded()
+			config.Spec.Scheduler.Replicas = ptr.To(tt.replicas)
+			config.Spec.Scheduler.Service.PodDisruptionBudget = &common.PodDisruptionBudget{
+				Enabled:        ptr.To(tt.pdbEnabled),
+				MaxUnavailable: ptr.To(tt.maxUnavailable),
+			}
+
+			s := NewSchedulerForShard(shard)
+			obj, err := s.podDisruptionBudgetForShard(ctx, client, config, shard)
+			require.NoError(t, err)
+
+			if !tt.expectPDBCreation {
+				assert.Nil(t, obj)
+				return
+			}
+
+			require.NotNil(t, obj)
+			pdb, ok := obj.(*policyv1.PodDisruptionBudget)
+			require.True(t, ok, "object should be PodDisruptionBudget")
+			assert.Equal(t, "kai-scheduler-default", pdb.Name)
+			assert.Equal(t, constants.DefaultKAINamespace, pdb.Namespace)
+			require.NotNil(t, pdb.Spec.MaxUnavailable)
+			assert.Equal(t, tt.maxUnavailable, pdb.Spec.MaxUnavailable.IntVal)
+			require.NotNil(t, pdb.Spec.Selector)
+			assert.Equal(t, "kai-scheduler-default", pdb.Spec.Selector.MatchLabels["app"])
+		})
+	}
+}

pkg/operator/operands/scheduler/scheduler.go

@@ -58,6 +58,7 @@ func (s *SchedulerForShard) DesiredState(
 	objects := []client.Object{}
 	for _, resourceFunc := range []resourceForShard{
 		s.deploymentForShard,
+		s.podDisruptionBudgetForShard,
 		s.configMapForShard,
 		s.serviceForShard,
 		s.endpointSliceForShard,

pkg/operator/operands/scheduler/scheduler_test.go

@@ -11,8 +11,10 @@ import (
 	. "github.com/onsi/gomega"
 
 	appsv1 "k8s.io/api/apps/v1"
+	policyv1 "k8s.io/api/policy/v1"
 
 	kaiv1 "github.com/kai-scheduler/KAI-scheduler/pkg/apis/kai/v1"
+	"github.com/kai-scheduler/KAI-scheduler/pkg/apis/kai/v1/common"
 
 	v1 "k8s.io/api/core/v1"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
@@ -130,6 +132,80 @@ var _ = Describe("Scheduler", func() {
 		Expect(len(services.Items)).To(Equal(2))
 	})
 
+	It("Should create one PDB per shard when HA and PDB enabled", func(ctx context.Context) {
+		kaiConfig.Spec.Scheduler.Replicas = ptr.To(int32(2))
+		kaiConfig.Spec.Scheduler.Service.PodDisruptionBudget = &common.PodDisruptionBudget{
+			Enabled:        ptr.To(true),
+			MaxUnavailable: ptr.To(int32(1)),
+		}
+
+		desiredState, err := schedulerOperandForShard.DesiredState(ctx, fakeClient, kaiConfig)
+		Expect(err).To(BeNil())
+		for _, obj := range desiredState {
+			Expect(fakeClient.Create(ctx, obj)).To(Succeed())
+		}
+
+		otherShard := &kaiv1.SchedulingShard{
+			ObjectMeta: metav1.ObjectMeta{Name: "other"},
+		}
+		otherShard.Spec.SetDefaultsWhereNeeded()
+		otherDesiredState, err := NewSchedulerForShard(otherShard).DesiredState(ctx, fakeClient, kaiConfig)
+		Expect(err).To(BeNil())
+		for _, obj := range otherDesiredState {
+			Expect(fakeClient.Create(ctx, obj)).To(Succeed())
+		}
+
+		pdbs := &policyv1.PodDisruptionBudgetList{}
+		Expect(fakeClient.List(ctx, pdbs, client.InNamespace(kaiConfig.Spec.Namespace))).To(Succeed())
+		Expect(pdbs.Items).To(HaveLen(2))
+		pdbNames := []string{pdbs.Items[0].Name, pdbs.Items[1].Name}
+		Expect(pdbNames).To(ConsistOf("kai-scheduler-default", "kai-scheduler-other"))
+	})
+
+	Context("PodDisruptionBudget DesiredState", func() {
+		It("includes PDB when HA and enabled", func(ctx context.Context) {
+			kaiConfig.Spec.Scheduler.Replicas = ptr.To(int32(2))
+			kaiConfig.Spec.Scheduler.Service.PodDisruptionBudget = &common.PodDisruptionBudget{
+				Enabled: ptr.To(true),
+			}
+
+			desiredState, err := schedulerOperandForShard.DesiredState(ctx, fakeClient, kaiConfig)
+			Expect(err).To(BeNil())
+			Expect(desiredState).To(HaveLen(5))
+
+			var pdbCount int
+			for _, obj := range desiredState {
+				if _, ok := obj.(*policyv1.PodDisruptionBudget); ok {
+					pdbCount++
+				}
+			}
+			Expect(pdbCount).To(Equal(1))
+		})
+
+		It("omits PDB when HA but disabled", func(ctx context.Context) {
+			kaiConfig.Spec.Scheduler.Replicas = ptr.To(int32(2))
+			kaiConfig.Spec.Scheduler.Service.PodDisruptionBudget = &common.PodDisruptionBudget{
+				Enabled: ptr.To(false),
+			}
+
+			desiredState, err := schedulerOperandForShard.DesiredState(ctx, fakeClient, kaiConfig)
+			Expect(err).To(BeNil())
+			Expect(desiredState).To(HaveLen(4))
+
+			for _, obj := range desiredState {
+				Expect(obj).NotTo(BeAssignableToTypeOf(&policyv1.PodDisruptionBudget{}))
+			}
+		})
+
+		It("returns empty desired state when scheduler disabled", func(ctx context.Context) {
+			kaiConfig.Spec.Scheduler.Service.Enabled = ptr.To(false)
+
+			desiredState, err := schedulerOperandForShard.DesiredState(ctx, fakeClient, kaiConfig)
+			Expect(err).To(BeNil())
+			Expect(desiredState).To(BeEmpty())
+		})
+	})
+
 	Context("ConfigMap", func() {
 		It("Should create configmap", func(ctx context.Context) {
 			s := NewSchedulerForShard(shard)