v0.14.8

Fixed

• Fixed extended resources present on only a subset of nodes being reported as unavailable cluster-wide: ResourceVector.SetMax now grows the accumulator to the longer vector's length instead of silently dropping resource indices discovered after the first-iterated node, which caused pods requesting such resources to be rejected as unschedulable ("No node in the node-pool has X resources") depending on node map iteration order. #1851
• Block NaN value for fraction in the pod admission #1798 davidLif
• In the fractional admission checks, check that the fractional value can be parsed as a quantity. #1798 davidLif
• Podgrouper now rejects negative PyTorch replica indexes and LWS worker indexes, and caps the number of subgroups created for block-level segmentation at 10000 to avoid unbounded PodGroup fan-out. davidLif
• Fixed GPU-sharing pods with dotted pod names generating invalid ConfigMap-backed volume names. Volume names are now sanitized to valid DNS labels while preserving original ConfigMap references used for shared-GPU injection.
• Fixed scheduler panic during reclaim when building eviction messages for jobs in root-level queues (ParentQueue empty) that reclaim across hierarchy branches. #1863
• Fixed GPU-sharing pods with dotted pod names generating invalid ConfigMap-backed volume names. Volume names are now sanitized to valid DNS labels while preserving original ConfigMap references used for shared-GPU injection. #1728
• Allow creating PodGroups with an explicit minMember of 0

v0.12.24

Fixed

• Block NaN value for fraction in the pod admission #1798 davidLif
• In the fractional admission checks, check that the fractional value can be parsed as a quantity. #1798 davidLif
• Fixed GPU-sharing pods with dotted pod names generating invalid ConfigMap-backed volume names. Volume names are now sanitized to valid DNS labels while preserving original ConfigMap references used for shared-GPU injection.
• Allow creating PodGroups with an explicit minMember of 0

v0.16.9

Added

• Reservation pods inherit fractional pod tolerations

Fixed

• Allow PodGroup minMember and minSubGroup of 0 for workloads with no gang requirement
• Propagate Kubernetes client QPS and burst settings to operator-managed controllers
• Podgrouper skips WorkloadRunner wrapper so wrapped workloads keep their gang grouping

v0.17.0

Added

• Added preemption delay: a kai.scheduler/preemption-delay pod annotation (or preemptionDelay PodGroup spec field, metav1.Duration format) defines a minimal pending time before a workload may trigger eviction of others via preempt, reclaim or consolidation — giving cluster autoscalers a window to provision nodes first. The window re-arms after each eviction (kai.scheduler/last-eviction-timestamp annotation); allocation into free capacity and the workload's own evictability are unaffected (docs, design). #1832
• Added global.resourceReservation.createServiceAccount Helm value (default true) to allow disabling creation of the resource-reservation ServiceAccount, for embedding KAI in a parent chart that creates the ServiceAccount itself.
• Added defaultPriorityClasses.enabled Helm value (default true) for installations that manage KAI PriorityClasses externally.
• Added GitOps/ArgoCD install support (guide): kaiConfig.render Helm value (default false) renders the kai-config Config CR inline as a tracked release resource (mutually exclusive with kaiConfigDeployer.enabled), openshift value (default false) forces OpenShift mode where lookup auto-detection is unavailable under offline rendering, and ArgoCD PostDelete hook and Prune=false annotations (requires ArgoCD >= 2.10). #1794 #1751
• Added topology level aliases: a Topology level may declare an alias (e.g. rack), usable in place of the raw node label key in a workload's requiredTopologyLevel/preferredTopologyLevel. Aliases are one-to-one (unique within the Topology and must not collide with a nodeLabel, enforced by a new Topology validating webhook) and may be edited freely (the levels immutability rule now freezes only the nodeLabel structure). When a level has no alias, behavior is unchanged and raw label keys keep working. #1498
• Added a Karta fallback podgrouper plugin that lets workload owners define gang-scheduling behavior declaratively via Karta definitions, without writing a native KAI plugin. The plugin translates a Karta gangScheduling.podGroup instruction into a KAI PodGroup with optional SubGroups and topology constraints; native KAI plugins take precedence, and the alpha podGroups instruction format remains supported for compatibility. #1877 davidLif
• Publish FIPS-enabled image variants (<version>-fips) for every release, built with the Go toolchain's native FIPS 140-3 mode (GOFIPS140), and added a global.fips Helm value (default false) that appends -fips to every resolved image tag (guide). #1867
• Added global.resourceReservation.createNamespace Helm value (default true) to allow disabling creation of the resource-reservation namespace, for embedding KAI in a parent chart that creates the namespace itself.
• Reservation pods inherit fractional pod tolerations
• Helm value global.priorityClassName sets a PriorityClass on all KAI control-plane pods
• NUMA-aware node scoring preferring fewest-NUMA-zone placement
• Added support for configuring scheduler Pod Disruption Budget via Helm values (scheduler.podDisruptionBudget) when running multiple replicas per scheduling shard. #1624 dttung2905
• override stale gang eviction grace period #1913 thebhdn
• Added support for configuring pod-grouper Pod Disruption Budget via Helm values (podgrouper.podDisruptionBudget) when running multiple replicas #1477 dttung2905
• Added support for DRA-backed extended resources (KEP-5004). Pods can request a DeviceClass's extendedResourceName without a ResourceClaim, and the scheduler routes it through DRA. See the design DRA-backed extended resources. #1943 gshaibi

Changed

• Reduced scheduler memory allocations and improved performance on clusters using NUMA topology alignment.
• make changelog accepts KIND and BODY vars for non-interactive use by agents
• Scenario search now skips re-simulating equivalent victim-set candidates that already failed simulation for the same pending job (reclaim, preempt, consolidation). Skipped candidates are recorded as state="duplicate" in scenario_search_scenarios_total. #1719
• Podgrouper now preserves an existing PodGroup's topology constraint when the workload does not specify one, so an externally-assigned topology is not overwritten. Workload topology annotations still take precedence when present.
• Removed unused queuecontroller.certSecretName and admission.certSecretName Helm values; webhook TLS secrets are created and managed by the operator (queue-webhook-tls-secret, kai-admission-webhook-tls-secret). #1791 dttung2905

Fixed

• Scenario search no longer leaks a rejected scenario's victim nodes into the probe's feasible-node set: the solver now rolls back feasible-node additions after validator-rejected and errored simulations, not only after cleanly unsolved ones. #1719
• Reduced scheduler memory use during large reclaim operations by removing redundant per-job-pair min-runtime protection caching; effective min-runtime durations remain cached per queue pair. #1808
• Podgrouper now rejects negative PyTorch replica indexes and LWS worker indexes, and caps the number of subgroups created for block-level segmentation at 10000 to avoid unbounded PodGroup fan-out. davidLif
• Fixed extended resources present on only a subset of nodes being reported as unavailable cluster-wide: ResourceVector.SetMax now grows the accumulator to the longer vector's length instead of silently dropping resource indices discovered after the first-iterated node, which caused pods requesting such resources to be rejected as unschedulable ("No node in the node-pool has X resources") depending on node map iteration order. #1851
• In the fractional admission checks, check that the fractional value can be parsed as a quantity. #1798 davidLif
• Scoped the operator's informer cache for Pods, Leases and EndpointSlices to the KAI namespace and stripped managed fields from cached objects. Since v0.15.0 the operator cached every such object in the cluster, so its memory grew with cluster size and exceeded the default 256Mi limit on large clusters. #1780
• Reduced transient scheduler allocations during large reclaim operations by comparing proportion queue state and cached resource vectors directly instead of repeatedly materializing resource maps.
• Fixed scheduler panic during reclaim when building eviction messages for jobs in root-level queues (ParentQueue empty) that reclaim across hierarchy branches. #1863
• Block NaN value for fraction in the pod admission #1798 davidLif
• Reduced allocation churn in the scheduler hot path: cached Schedulable() result as a package-level singleton and lazily formatted logNodeSetsPluginResult node names only when verbose logging is enabled.
• Use the maximum gpu size ine the cluster rather then the minimum when checking a potential overLimit or isNonPreemptebleOverquota for a pod. #1792 davidLif
• Fix the MinNodeGPUMemoryMiB calculation in the scheduler. This affected allocations for fractional pod requesting gpu "gpu-memory". #1792 davidLif
• Scheduler cache now filters terminal Pods at watch time to reduce memory use, while still watching Pods bound by other schedulers so their resource usage is counted in allocatable calculations. #1645 enoodle
• Saturated the DRA GPU device-count accumulation so a ResourceClaim requesting an oversized device count can no longer overflow the queue controller's int64 GPU total to a negative value. #1873 thc1006
• Clean deleted Pod status updates
• Prevent some scheduler OOM kills by setting Go's memory limit
• Clear stale PodGroup UnschedulableOnNodePool conditions after workloads schedule.
• NUMA plugin no longer counts non-integral container CPU toward NUMA alignment
• NUMA plugin now aligns non-Guaranteed pods that request GPUs, matching the kubelet's device manager (device alignment is QoS-independent).
• ...

v0.16.8

Fixed

• Guard custom scalingpod namespaces and resource-reservation ServiceAccounts in the Helm chart #1733 dttung2905

v0.16.7

Fixed

• Segmented elastic PyTorchJob now respects minReplicas instead of requiring all worker segments

v0.16.6

Fixed

• Segmented PyTorch and LWS PodGrouper now uses minSubGroup on parent SubGroups, fixing admission webhook rejection. #1927

v0.16.5

Changed

• Reject negative PyTorch replica and LWS worker indexes, and cap block-level segmentation at 10,000 subgroups to prevent unbounded PodGroup fan-out. #1816 davidLif

Fixed

• Fixed extended resources present on only a subset of nodes being reported as unavailable cluster-wide: ResourceVector.SetMax now grows the accumulator to the longer vector's length instead of silently dropping resource indices discovered after the first-iterated node, which caused pods requesting such resources to be rejected as unschedulable ("No node in the node-pool has X resources") depending on node map iteration order. #1851
• Scheduler snapshot now correctly captures the plugin configuration even when /get-snapshot is requested between scheduling cycles (previously the config field was written as null, causing snapshot-tool to panic on replay). #1885
• Fixed scheduler panic during reclaim when building eviction messages for jobs in root-level queues (ParentQueue empty) that reclaim across hierarchy branches. #1863
• Sanitize GPU-sharing volume names for pods with dots while preserving their ConfigMap references. #1728

v0.16.4

What's Changed

Added

• Publish FIPS-enabled image variants (<version>-fips) for every release, built with the Go toolchain's native FIPS 140-3 mode (GOFIPS140), and added a global.fips Helm value (default false) that appends -fips to every resolved image tag (guide). #1867
• Added global.nodePoolLabelKey Helm value to configure spec.global.nodePoolLabelKey in the Config CR for KAI sharding #1774.

Fixed

• Scoped the operator's informer cache for Pods, Leases and EndpointSlices to the KAI namespace and stripped managed fields from cached objects. Since v0.15.0 the operator cached every such object in the cluster, so its memory grew with cluster size and exceeded the default 256Mi limit on large clusters. #1780
• Block NaN value for fraction in the pod admission #1798 davidLif
• In the fractional admission checks, check that the fractional value can be parsed as a quantity. #1798 davidLif

Full Changelog: v0.16.3...v0.16.4

v0.14.7

What's Changed

Added

• For the pytorch and lws plugins, validate workerindex > 0 and cap block-level segmentation size by @KaiPilotBot in #1829

Fixed

• Block NaN value for fraction in the pod admission - master by @KaiPilotBot in #1802
• Remove unnecessary NaN check - v0.14 by @davidLif in #1813

Full Changelog: v0.14.6...v0.14.7