v0.15.0

What's Changed

Added

• Added enabled Helm values for binder, podgrouper, podgroupcontroller, queuecontroller, admission, and scheduler to allow disabling individual components from values.yaml. Previously these were hardcoded to true in the kai-config template.
• Added prometheus.enabled and prometheus.externalPrometheusUrl Helm values to configure Prometheus from values.yaml #907
• Added validation for subgroup name in podgroup faizanexe
• Added memory profile and run duration to snapshot tool #1411
• Added support for configuring pod and container security contexts on resource reservation pods via CLI flags AdheipSingh
• Added operator.logLevel Helm value to configure the operator log level (maps to --zap-log-level when set) #1446 dttung2905
• The scheduler now implements elastic PodGroups on both the subgroup level (minSubGroup) and pods (minAvailable). This allows for elasticity on all of the podgroup tree hierarchy. #1416 - davidLif
• Allow the configuration of plugins in the binder service. #1480 - davidLif
• Added support for configuring scheduler log level and custom scheduler args via Helm values (scheduler.args) #1452 dttung2905
• Added global.jsonLog Helm value to enable JSON-formatted logging for use with log aggregation platforms
• Added crdupgrader.image.registry Helm value to override global.registry for the crd-upgrader pre-install/pre-upgrade hook image, allowing the hook image to be served from a separate mirror without redirecting all chart images. #1404
• Added support for externally-created PodGroups. Workloads can opt out of podgrouper mutation with kai.scheduler/skip-podgrouper: "true" on the pod or owner chain, join an existing PodGroup via pod-group-name, and now get a pod condition when they reference a non-existent subgroup. #1420
• Added --stuck-in-releasing-threshold scheduler flag (default 2m) controlling how long a Running pod with a deletionTimestamp remains classified as Releasing before being reclassified as StuckInReleasing and excluded from pipelining. Configurable per shard via SchedulingShard.spec.args.stuck-in-releasing-threshold.

Changed

• Breaking: JobSet PodGroups no longer auto-calculate minAvailable from parallelism × replicas. The default is now 1. Use the kai.scheduler/batch-min-member annotation to set a custom value.
• Bumped k8s.io/* module group from v0.34.x to v0.35.4, k8s.io/kubernetes to v1.35.4, and sigs.k8s.io/controller-runtime to v0.23.3, enabling KEP-4671 Workload API types. #1466
• Rebuilt the crd-upgrader hook image on alpine:3.20 instead of ubi9/ubi-minimal. Image size drops from ~165 MB to ~67 MB uncompressed (~60% reduction), shrinking cold-pull latency on ephemeral CI runners. The image is also reused by the topology-migration and post-delete hook jobs as a generic kubectl + bash toolbox, so bash is preserved on the runtime image. #1404

Fixed

• Streaming snapshot JSON directly into the zip writer to avoid OOM on large clusters. The /get-snapshot endpoint previously buffered the entire JSON payload in memory (~3x the data size); it now streams per-element, reducing peak memory to ~1x. #1564
• Fixed additionalImagePullSecrets in Config CR rendering as map[name:...] instead of plain strings by extracting .name from global.imagePullSecrets objects. Also propagated global.imagePullSecrets to all Helm hook jobs (crd-upgrader, topology-migration, post-delete-cleanup)
• Added global.nodeSelector, global.tolerations, global.affinity, global.securityContext support to the post-delete job hook.
• Fixed Helm template writing imagesPullSecret (string) instead of additionalImagePullSecrets (array) in Config CR, causing image pull secrets to be silently ignored. Added backward-compatible deprecated imagesPullSecret field to CRD schema. #942
• Fixed windowSize field in SchedulingShard CR to support Prometheus duration format (e.g. 1w, 7d). Previously, using windowSize: 1w as shown in the documentation caused the kai-operator to crash-loop with time: unknown unit "w" in duration "1w".
• Race condition where SyncForGpuGroup could prematurely delete reservation pods when the informer cache had not yet propagated GPU group labels on recently-bound fraction pods. The binder now checks for active BindRequests referencing the GPU group before deleting a reservation pod.
• Fixed non-preemptible multi-device GPU memory jobs being allowed to exceed their queue's deserved GPU quota. The per-node quota check now correctly accounts for all requested GPU devices. #1369
• Added resourceclaims/binding RBAC permission to the binder ClusterRole for compatibility with Kubernetes v1.36+, where the DRAResourceClaimGranularStatusAuthorization feature gate requires explicit permission on the resourceclaims/binding subresource to modify status.allocation and status.reservedFor on ResourceClaims. #1372 praveen0raj
• Allow users to override minMember for k8s batch Jobs and JobSets using the kai.scheduler/batch-min-member annotation #1308 itsomri
• Fixed a bug where nil minMember caused subgroups creation to fail in scheduler #1407 itsomri
• Improved performance by evaluating SetNode once per session instead of on each predicate evaluation #1421 itsomri
• Added persistent volumes to cluster snapshot #1424 itsomri
• Improved scheduling performance for preempt/reclaim/consolidate actions on jobs with many tasks by replacing per-task linear probing with exponential+binary search in the job solver, reducing the number of scenario simulations from O(n) to O(log n) #1435 itsomri
• Avoid expensive solver-backed reclaim/preempt/consolidation work for jobs already blocked by victim-invariant pre-solver failures such as missing PVCs, missing required ConfigMaps, or requests larger than the maximum node size. #1502
• Fixed skipTopOwnerGrouper not propagating per-type defaults (priority class and preemptibility) for skipped owners (e.g. DynamoGraphDeployment), causing PodGroup spec to retain stale values after defaults ConfigMap updates.
• Fixed binder DRA detection on clusters where the upstream DynamicResourceAllocation feature gate does not reflect server-side DRA availability. The binder now probes the API server during init (matching the scheduler) so the DRA plugin is gated on the same authoritative decision. #1481
• Suppressed noisy Reconciler error logs and PodGrouperWarning events on transient PodGroup update conflicts. The podgrouper now treats IsConflict errors as expected and silently requeues the reconcile instead of surfacing the apiserver's "object has been modified" message.
• Fixed kai-operator not reconciling on Prometheus and ServiceMonitor changes. The Config controller now watches owned Prometheus and ServiceMonitor resources, so deletions and drift trigger reconciliation. CRD presence is checked at startup against the API server (the scheme-only check used previously could not detect missing CRDs), and the watch is registered only when the CRDs are installed. #877
• Added before-hook-creation to the crd-upgrader Helm hook delete policy so failed hook Jobs no longer block subsequent helm upgrade --install retries. Aligns with the policy already used by the chart's other hook resources. #1404
• Fixed kai-operator leader-election event emission by adding RBAC permission for core events (create, patch, update) so operators can publish leadership events instead of logging events is forbidden. #1572 dttung2905
• The scheduler's per-shard Service is now populated by an operator-managed EndpointSlice pointing at the current leader-election Lease holder, which is connected to the service of the shard's scheduler. This allows the service to route all it's incoming request to the lease-holding pod of the scheduler deployment. #1593 davidLif
• Fixed podgroupcontroller logging spurious errors on every reconcile for completed/failed pods because it tried to fetch DRA ResourceClaim objects that the DRA driver had already deleted. Terminal pods now skip the ResourceClaim lookup entirely, mirroring the scheduler-side fix in #1456. #1529

New Contributors

• @JheSue made their first contribution in #1371
• @praveen0raj made their first contribution in #1372
• @Dave-London made their first contribution in #976
• @mfchanou made their first contribution in #1368
• @vrv3814 made their first contribution in #1385
• @kube-gopher made their first contribution in #1440
• @yysindi made their first contribution in #1564

Full Changelog: v0.14.0...v0.15.0