Skip to content

Commit c6e8ce6

Browse files
SiorMeirSiorMeir
committed
feat: add kai.scheduler/v1 group, usagedb structs, podgrouper constants
Signed-off-by: SiorMeir <msior@nvidia.com>
1 parent 1702de7 commit c6e8ce6

62 files changed

Lines changed: 17485 additions & 41 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.changes/unreleased/Added-20260723-140000.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
kind: Added
2+
body: |-
3+
`kai.scheduler/v1` API group (`kai/v1`): `Config` and `SchedulingShard` types plus their component sub-packages, with generated CRD manifests. `usagedb` config structs (`UsageDBConfig`, `UsageParams`, `WindowType`), `podgrouper/constants`, and `utilities/math` (`SaturatingAdd`).

‎.changes/unreleased/Changed-20260723-140100.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
kind: Changed
2+
body: |-
3+
Re-synced `utilities/resources` (DRA GPU-count overflow fix), `scheduling/v2alpha2` PodGroup types, and `podgrouper/constants` with KAI-Scheduler main.

‎config/crd/kai.scheduler_configs.yaml‎

Lines changed: 11745 additions & 0 deletions
Large diffs are not rendered by default.

‎config/crd/kai.scheduler_schedulingshards.yaml‎

Lines changed: 301 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,301 @@
1+
# Copyright 2025 NVIDIA CORPORATION
2+
# SPDX-License-Identifier: Apache-2.0
3+
#
4+
# DO NOT EDIT - This file is auto-generated by controller-gen
5+
# To modify RBAC permissions, edit the +kubebuilder:rbac markers in the source code
6+
# and run 'make manifests' to regenerate this file.
7+
---
8+
apiVersion: apiextensions.k8s.io/v1
9+
kind: CustomResourceDefinition
10+
metadata:
11+
annotations:
12+
controller-gen.kubebuilder.io/version: v0.20.1
13+
name: schedulingshards.kai.scheduler
14+
spec:
15+
group: kai.scheduler
16+
names:
17+
kind: SchedulingShard
18+
listKind: SchedulingShardList
19+
plural: schedulingshards
20+
singular: schedulingshard
21+
scope: Cluster
22+
versions:
23+
- name: v1
24+
schema:
25+
openAPIV3Schema:
26+
description: SchedulingShard is the Schema for the schedulingshards API
27+
properties:
28+
apiVersion:
29+
description: |-
30+
APIVersion defines the versioned schema of this representation of an object.
31+
Servers should convert recognized schemas to the latest internal value, and
32+
may reject unrecognized values.
33+
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources
34+
type: string
35+
kind:
36+
description: |-
37+
Kind is a string value representing the REST resource this object represents.
38+
Servers may infer this from the endpoint the client submits requests to.
39+
Cannot be updated.
40+
In CamelCase.
41+
More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds
42+
type: string
43+
metadata:
44+
type: object
45+
spec:
46+
description: SchedulingShardSpec defines the desired state of SchedulingShard
47+
properties:
48+
actions:
49+
additionalProperties:
50+
description: ActionConfig allows overriding action settings in the
51+
scheduler configuration.
52+
properties:
53+
enabled:
54+
description: Enabled controls whether this action is active.
55+
Defaults to true.
56+
type: boolean
57+
priority:
58+
description: |-
59+
Priority controls the ordering of this action. Higher values run first.
60+
Built-in actions use priorities in the range 0-10000, spaced by 100.
61+
type: integer
62+
type: object
63+
description: |-
64+
Actions allows overriding action configuration. Keys are action names.
65+
Built-in actions can be disabled or reordered.
66+
New actions can be added by specifying a name not in the default set.
67+
Default actions and their priorities (higher runs first):
68+
allocate=500, consolidation=400, reclaim=300, preempt=200, stalegangeviction=100.
69+
type: object
70+
args:
71+
additionalProperties:
72+
type: string
73+
description: |-
74+
Args specifies custom CLI arguments for the scheduler. These are merged with automatically generated flags.
75+
Valid flags are those defined in the scheduler's code. Usage examples:
76+
- To pass "--custom-flag=value": Args: {"custom-flag": "value"}
77+
- To override default flags like "--leader-elect": Args: {"leader-elect": "true"}
78+
Note:
79+
* Flag names must match exactly (case-sensitive)
80+
* Values are passed as strings
81+
* Only valid flags defined in the scheduler's flag set will be accepted
82+
* Duplicated flags will override the behavior of flags generated by other fields
83+
type: object
84+
kValue:
85+
description: KValue specifies the kValue for the proportion plugin.
86+
Default is 1.0.
87+
type: number
88+
minRuntime:
89+
description: MinRuntime specifies the minimum runtime of a jobs in
90+
the shard
91+
properties:
92+
preemptMinRuntime:
93+
description: PreemptMinRuntime specifies the minimum runtime of
94+
a job in queue before it can be preempted
95+
type: string
96+
reclaimMinRuntime:
97+
description: ReclaimMinRuntime specifies the minimum runtime of
98+
a job in queue before it can be reclaimed
99+
type: string
100+
type: object
101+
partitionLabelValue:
102+
description: PartitionLabelValue is the value for the partition label
103+
type: string
104+
placementStrategy:
105+
description: PlacementStrategy is the placement scheduler strategy
106+
properties:
107+
cpu:
108+
description: CPU scheduling strategy (binpack/spread)
109+
type: string
110+
gpu:
111+
description: GPU scheduling strategy (binpack/spread)
112+
type: string
113+
type: object
114+
plugins:
115+
additionalProperties:
116+
description: PluginConfig allows overriding plugin settings in the
117+
scheduler configuration.
118+
properties:
119+
arguments:
120+
additionalProperties:
121+
type: string
122+
description: |-
123+
Arguments are key-value pairs passed to the plugin. When specified, they fully replace
124+
the default arguments for the plugin.
125+
type: object
126+
enabled:
127+
description: Enabled controls whether this plugin is active.
128+
Defaults to true.
129+
type: boolean
130+
priority:
131+
description: |-
132+
Priority controls the ordering of this plugin. Higher values run first.
133+
Built-in plugins use priorities in the range 0-10000, spaced by 100.
134+
type: integer
135+
type: object
136+
description: |-
137+
Plugins allows overriding plugin configuration. Keys are plugin names.
138+
Built-in plugins can be disabled, reordered, or have their arguments changed.
139+
New plugins can be added by specifying a name not in the default set.
140+
Default plugins and their priorities (higher runs first):
141+
predicates=1900, proportion=1800, priority=1700, nodeavailability=1600,
142+
resourcetype=1500, podaffinity=1400, elastic=1300, kubeflow=1200,
143+
ray=1100, subgrouporder=1000, taskorder=900, nominatednode=800,
144+
dynamicresources=700, minruntime=600, topology=500, snapshot=400,
145+
sg-nodelocalgreedy=360, sg-multinodegang=350, gpupack/gpuspread=300,
146+
nodeplacement=200, gpusharingorder=100.
147+
type: object
148+
queueDepthPerAction:
149+
additionalProperties:
150+
type: integer
151+
description: QueueDepthPerAction max number of jobs to try for action
152+
per queue
153+
type: object
154+
scenarioSearchBudgets:
155+
description: ScenarioSearchBudgets configures alpha/experimental time
156+
budgets for scenario search.
157+
properties:
158+
maxActionSearchDuration:
159+
additionalProperties:
160+
type: string
161+
description: |-
162+
MaxActionSearchDuration limits total scenario search time per scheduler action.
163+
Keys are action names, with "default" used as the fallback budget.
164+
type: object
165+
maxGeneratorSearchDuration:
166+
additionalProperties:
167+
type: string
168+
description: |-
169+
MaxGeneratorSearchDuration limits scenario search time per generator attempt.
170+
Keys are generator names, with "default" used as the fallback budget.
171+
type: object
172+
maxJobSearchDuration:
173+
description: MaxJobSearchDuration limits total scenario search
174+
time per pending job.
175+
type: string
176+
minJobSearchDuration:
177+
description: |-
178+
MinJobSearchDuration guarantees each pending job this much scenario search time
179+
before action and generator budgets can stop the job's search.
180+
type: string
181+
type: object
182+
usageDBConfig:
183+
description: UsageDBConfig defines configuration for the usage db
184+
client
185+
properties:
186+
clientType:
187+
type: string
188+
connectionString:
189+
type: string
190+
connectionStringEnvVar:
191+
type: string
192+
usageParams:
193+
description: UsageParams defines common params for all usage db
194+
clients. Some clients may not support all the params.
195+
properties:
196+
cronString:
197+
description: The cron string defining the behavior of the
198+
cron window.
199+
type: string
200+
extraParams:
201+
additionalProperties:
202+
type: string
203+
description: ExtraParams are extra parameters for the usage
204+
db client, which are client specific.
205+
type: object
206+
fetchInterval:
207+
description: Fetch interval of the usage. Default is 1 minute.
208+
type: string
209+
halfLifePeriod:
210+
description: Half life period of the usage. If not set, or
211+
set to 0, the usage will not be decayed.
212+
type: string
213+
stalenessPeriod:
214+
description: Staleness period of the usage. Default is 5 minutes.
215+
type: string
216+
tumblingWindowStartTime:
217+
description: The start timestamp of the tumbling window. If
218+
not set, defaults to the current time.
219+
format: date-time
220+
type: string
221+
waitTimeout:
222+
description: Wait timeout of the usage. Default is 1 minute.
223+
type: string
224+
windowSize:
225+
description: Window size of the usage. Default is 1 week.
226+
pattern: ^(0|(([0-9]+)y)?(([0-9]+)w)?(([0-9]+)d)?(([0-9]+)h)?(([0-9]+)m)?(([0-9]+)s)?(([0-9]+)ms)?)$
227+
type: string
228+
windowType:
229+
description: Window type for time-series aggregation. If not
230+
set, defaults to sliding.
231+
type: string
232+
type: object
233+
required:
234+
- clientType
235+
type: object
236+
type: object
237+
status:
238+
description: SchedulingShardStatus defines the observed state of SchedulingShard
239+
properties:
240+
conditions:
241+
items:
242+
description: Condition contains details for one aspect of the current
243+
state of this API Resource.
244+
properties:
245+
lastTransitionTime:
246+
description: |-
247+
lastTransitionTime is the last time the condition transitioned from one status to another.
248+
This should be when the underlying condition changed. If that is not known, then using the time when the API field changed is acceptable.
249+
format: date-time
250+
type: string
251+
message:
252+
description: |-
253+
message is a human readable message indicating details about the transition.
254+
This may be an empty string.
255+
maxLength: 32768
256+
type: string
257+
observedGeneration:
258+
description: |-
259+
observedGeneration represents the .metadata.generation that the condition was set based upon.
260+
For instance, if .metadata.generation is currently 12, but the .status.conditions[x].observedGeneration is 9, the condition is out of date
261+
with respect to the current state of the instance.
262+
format: int64
263+
minimum: 0
264+
type: integer
265+
reason:
266+
description: |-
267+
reason contains a programmatic identifier indicating the reason for the condition's last transition.
268+
Producers of specific condition types may define expected values and meanings for this field,
269+
and whether the values are considered a guaranteed API.
270+
The value should be a CamelCase string.
271+
This field may not be empty.
272+
maxLength: 1024
273+
minLength: 1
274+
pattern: ^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$
275+
type: string
276+
status:
277+
description: status of the condition, one of True, False, Unknown.
278+
enum:
279+
- "True"
280+
- "False"
281+
- Unknown
282+
type: string
283+
type:
284+
description: type of condition in CamelCase or in foo.example.com/CamelCase.
285+
maxLength: 316
286+
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
287+
type: string
288+
required:
289+
- lastTransitionTime
290+
- message
291+
- reason
292+
- status
293+
- type
294+
type: object
295+
type: array
296+
type: object
297+
type: object
298+
served: true
299+
storage: true
300+
subresources:
301+
status: {}

‎config/crd/scheduling.run.ai_podgroups.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,12 @@ spec:
101101
-1 and 1
102102
format: int32
103103
type: integer
104+
stalenessGracePeriod:
105+
description: |-
106+
StalenessGracePeriod is the minimum duration a stale PodGroup it allowed to remain in stale
107+
status before stale workloads may be evicted to make room. Negative values disable stale gang
108+
eviction for this PodGroup. Defaults to the scheduler's global staleness grace period.
109+
type: string
104110
subGroups:
105111
description: SubGroups defines finer-grained subsets of pods within
106112
the PodGroup with individual scheduling constraints

‎go.mod‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,12 @@ go 1.26.3
55
require (
66
github.com/onsi/ginkgo/v2 v2.28.1
77
github.com/onsi/gomega v1.39.1
8+
github.com/prometheus-operator/prometheus-operator/pkg/apis/monitoring v0.88.0
89
github.com/stretchr/testify v1.11.1
910
golang.org/x/exp v0.0.0-20260218203240-3dfff04db8fa
1011
k8s.io/api v0.35.4
1112
k8s.io/apimachinery v0.35.4
13+
k8s.io/autoscaler/vertical-pod-autoscaler v1.5.1
1214
k8s.io/client-go v0.35.4
1315
k8s.io/dynamic-resource-allocation v0.35.4
1416
k8s.io/kubernetes v1.35.4
@@ -20,15 +22,15 @@ require (
2022
github.com/Masterminds/semver/v3 v3.4.0 // indirect
2123
github.com/beorn7/perks v1.0.1 // indirect
2224
github.com/cespare/xxhash/v2 v2.3.0 // indirect
23-
github.com/davecgh/go-spew v1.1.1 // indirect
24-
github.com/emicklei/go-restful/v3 v3.12.2 // indirect
25+
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
26+
github.com/emicklei/go-restful/v3 v3.13.0 // indirect
2527
github.com/evanphx/json-patch/v5 v5.9.11 // indirect
2628
github.com/fsnotify/fsnotify v1.9.0 // indirect
2729
github.com/fxamacker/cbor/v2 v2.9.0 // indirect
2830
github.com/go-logr/logr v1.4.3 // indirect
29-
github.com/go-openapi/jsonpointer v0.21.0 // indirect
30-
github.com/go-openapi/jsonreference v0.20.2 // indirect
31-
github.com/go-openapi/swag v0.23.0 // indirect
31+
github.com/go-openapi/jsonpointer v0.21.2 // indirect
32+
github.com/go-openapi/jsonreference v0.21.0 // indirect
33+
github.com/go-openapi/swag v0.23.1 // indirect
3234
github.com/go-task/slim-sprig/v3 v3.0.0 // indirect
3335
github.com/google/btree v1.1.3 // indirect
3436
github.com/google/gnostic-models v0.7.0 // indirect
@@ -37,15 +39,15 @@ require (
3739
github.com/google/uuid v1.6.0 // indirect
3840
github.com/josharian/intern v1.0.0 // indirect
3941
github.com/json-iterator/go v1.1.12 // indirect
40-
github.com/mailru/easyjson v0.7.7 // indirect
42+
github.com/mailru/easyjson v0.9.0 // indirect
4143
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
4244
github.com/modern-go/reflect2 v1.0.3-0.20250322232337-35a7c28c31ee // indirect
4345
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
44-
github.com/pmezard/go-difflib v1.0.0 // indirect
46+
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 // indirect
4547
github.com/prometheus/client_golang v1.23.2 // indirect
4648
github.com/prometheus/client_model v0.6.2 // indirect
4749
github.com/prometheus/common v0.66.1 // indirect
48-
github.com/prometheus/procfs v0.16.1 // indirect
50+
github.com/prometheus/procfs v0.17.0 // indirect
4951
github.com/spf13/pflag v1.0.9 // indirect
5052
github.com/x448/float16 v0.8.4 // indirect
5153
go.yaml.in/yaml/v2 v2.4.3 // indirect
@@ -57,7 +59,7 @@ require (
5759
golang.org/x/sys v0.41.0 // indirect
5860
golang.org/x/term v0.40.0 // indirect
5961
golang.org/x/text v0.34.0 // indirect
60-
golang.org/x/time v0.9.0 // indirect
62+
golang.org/x/time v0.12.0 // indirect
6163
golang.org/x/tools v0.42.0 // indirect
6264
gomodules.xyz/jsonpatch/v2 v2.4.0 // indirect
6365
google.golang.org/protobuf v1.36.8 // indirect

0 commit comments

Comments
 (0)