Skip to content

Commit 2914d32

Browse files
enoodleenoodle
authored
docs(operator): add resource sizing guide and calculator (#2026)
Signed-off-by: Erez Freiberger <enoodle@gmail.com>
1 parent 7ed7324 commit 2914d32

10 files changed

Lines changed: 2260 additions & 0 deletions

File tree

‎.github/workflows/deploy-resource-sizing-page.yaml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Copyright 2026 NVIDIA CORPORATION
2+
# SPDX-License-Identifier: Apache-2.0
3+
4+
name: Deploy Resource Sizing Calculator
5+
6+
on:
7+
push:
8+
branches: [main]
9+
paths:
10+
- "docs/resource-sizing/**"
11+
- "docs/assets/kai-logo-dark.png"
12+
- "docs/assets/kai-logo-light.png"
13+
- ".github/workflows/deploy-resource-sizing-page.yaml"
14+
workflow_dispatch:
15+
16+
permissions:
17+
contents: write
18+
19+
concurrency:
20+
group: pages
21+
cancel-in-progress: false
22+
23+
jobs:
24+
deploy:
25+
name: Deploy to GitHub Pages
26+
runs-on: ubuntu-latest
27+
28+
steps:
29+
- name: Checkout
30+
uses: actions/checkout@v7
31+
32+
- name: Test calculator
33+
run: node --test docs/resource-sizing/*.test.js
34+
35+
- name: Prepare site
36+
run: |
37+
cp -R docs/resource-sizing _site
38+
cp docs/assets/kai-logo-dark.png _site/kai-logo-dark.png
39+
cp docs/assets/kai-logo-light.png _site/kai-logo-light.png
40+
41+
- name: Deploy to gh-pages branch
42+
uses: peaceiris/actions-gh-pages@v4
43+
with:
44+
github_token: ${{ secrets.GITHUB_TOKEN }}
45+
publish_dir: _site
46+
publish_branch: gh-pages
47+
destination_dir: resource-sizing
48+
exclude_assets: '*.test.js'

‎docs/operator/README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -70,6 +70,7 @@ spec:
7070
```
7171

7272
- [Scheduling Shards](./scheduling-shards.md) - Advanced cluster partitioning
73+
- [Resource sizing](./resource-sizing.md) - Requests, limits, and VPA guidance for large clusters
7374

7475
## Logging
7576

‎docs/operator/resource-sizing.md‎

Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# Resource sizing
2+
3+
KAI Scheduler resource usage depends on cluster size and workload shape. A
4+
cluster with fewer nodes can require more scheduler memory when it has more
5+
Pods, larger jobs, or a larger submission backlog.
6+
7+
This guide provides tested starting profiles and sizing guidance for KAI
8+
Scheduler services.
9+
10+
## Quick start
11+
12+
1. Estimate the values below for each important workload scenario.
13+
2. Use a tested profile when the complete scenario fits its envelope.
14+
3. Otherwise, calculate the scheduler memory limit and use the larger result.
15+
4. Run a representative workload lifecycle before production rollout.
16+
17+
Do not combine unrelated maxima. For example, calculate a many-small-jobs
18+
scenario separately from a single-large-job scenario, then use the largest
19+
recommendation.
20+
21+
### Information to collect
22+
23+
| Input | Meaning |
24+
| --- | --- |
25+
| Nodes | Nodes usable by this scheduler shard |
26+
| Total Pods | Peak cluster-wide Pod objects, including non-KAI Pods |
27+
| Workloads | Peak simultaneously active KAI workloads |
28+
| Average workload size | Average schedulable Pods per workload |
29+
| Largest workload | Schedulable Pods in the largest workload |
30+
| Total GPUs | GPUs available to these workloads |
31+
| GPUs per worker Pod | Average GPU request per GPU worker Pod; use `1` when unsure |
32+
| Eligible nodes | Nodes where these workloads can run; use all nodes when unsure |
33+
34+
For CPU-only workloads, replace GPU capacity with the estimated maximum number
35+
of worker Pods that can run concurrently.
36+
37+
## Starting profiles
38+
39+
Values are `request / limit`. `omit` means that no CPU limit should be set
40+
until an uncapped load test establishes actual demand.
41+
42+
| Service | 500 CPU | 500 memory | 1000 CPU | 1000 memory |
43+
| --- | ---: | ---: | ---: | ---: |
44+
| Scheduler | `2 / 4` | `4Gi / 7Gi` | `3 / 5` | `7Gi / 8Gi` |
45+
| Binder | `250m / 1` | `3Gi / 4Gi` | `1 / 2` | `5Gi / 6Gi` |
46+
| Pod grouper | `250m / 1` | `1500Mi / 2Gi` | `500m / 2` | `3Gi / 4Gi` |
47+
| PodGroup controller | `500m / omit` | `2Gi / 3Gi` | `1 / omit` | `3500Mi / 4Gi` |
48+
| Queue controller | `250m / omit` | `256Mi / 512Mi` | `500m / omit` | `400Mi / 512Mi` |
49+
| Admission, per replica | `50m / 250m` | `64Mi / 128Mi` | `50m / 250m` | `64Mi / 128Mi` |
50+
| Operator | `25m / 100m` | `128Mi / 256Mi` | `25m / 100m` | `128Mi / 256Mi` |
51+
52+
The profiles were exercised across these workload shapes. The values are
53+
separate lifecycle maxima and did not all occur simultaneously.
54+
55+
| Workload property | 500 profile | 1000 profile |
56+
| --- | ---: | ---: |
57+
| Nodes | 520 | 1,008 |
58+
| Total Pods | 46,000 | 90,000 |
59+
| Active workloads | 8,000 | 16,000 |
60+
| Largest workload | 500 Pods | 1,000 Pods |
61+
| Average workload in the largest burst | 57 Pods | 102 Pods |
62+
| GPUs | 4,000 | 8,000 |
63+
64+
Use the next profile or the calculator when an expected scenario is larger.
65+
These values are starting points, not capacity guarantees. Pod shape, storage,
66+
DRA, topology constraints, GPU sharing, and enabled plugins can change resource
67+
usage.
68+
69+
## Calculate scheduler memory
70+
71+
Use the [resource sizing calculator](https://kai-scheduler.github.io/KAI-Scheduler/resource-sizing/)
72+
to get initial resources for the Scheduler, Binder, and controllers. It also
73+
generates a command that patches Config-managed services. The calculator runs
74+
entirely in the browser and does not send cluster information anywhere.
75+
76+
For the formula, assumptions, pressure tiers, and a worked example, see the
77+
[scheduler memory sizing deep dive](./scheduler-memory-sizing.md).
78+
79+
## Size Binder and controllers
80+
81+
Use these conservative initial memory requests. Round up and keep the selected
82+
profile's limit until a complete-cycle test supports changing it.
83+
84+
```text
85+
Binder:
86+
256Mi + max(50Mi * total Pods / 1000,
87+
100Mi * outstanding BindRequests / 1000)
88+
89+
Pod grouper:
90+
256Mi + 25Mi * retained KAI Pods / 1000
91+
92+
PodGroup controller:
93+
256Mi + 35Mi * retained KAI Pods / 1000
94+
95+
Queue controller:
96+
64Mi + 20Mi * PodGroups / 1000
97+
```
98+
99+
The calculator rounds formula results up to 128 MiB, never below the selected
100+
profile request. When a formula raises a request, it preserves the profile's
101+
request-to-limit headroom.
102+
103+
Use the deep dive's [`workloadPods`](./scheduler-memory-sizing.md#infer-workload-pods-and-capacity)
104+
as an initial BindRequest upper bound when no measurement is available. Do not
105+
add allowances for multiple dimensions that describe the same object
106+
population.
107+
108+
## Configure resources
109+
110+
The following example applies the tested 1000-node profile:
111+
112+
```yaml
113+
scheduler:
114+
resources:
115+
requests: {cpu: "3", memory: 7Gi}
116+
limits: {cpu: "5", memory: 8Gi}
117+
binder:
118+
resources:
119+
requests: {cpu: "1", memory: 5Gi}
120+
limits: {cpu: "2", memory: 6Gi}
121+
podgrouper:
122+
resources:
123+
requests: {cpu: 500m, memory: 3Gi}
124+
limits: {cpu: "2", memory: 4Gi}
125+
podgroupcontroller:
126+
resources:
127+
requests: {cpu: "1", memory: 3500Mi}
128+
limits: {memory: 4Gi}
129+
queuecontroller:
130+
resources:
131+
requests: {cpu: 500m, memory: 400Mi}
132+
limits: {memory: 512Mi}
133+
admission:
134+
resources:
135+
requests: {cpu: 50m, memory: 64Mi}
136+
limits: {cpu: 250m, memory: 128Mi}
137+
operator:
138+
resources:
139+
requests: {cpu: 25m, memory: 128Mi}
140+
limits: {cpu: 100m, memory: 256Mi}
141+
```
142+
143+
Resources can also be managed through `spec.<service>.service.resources` on the
144+
Config custom resource. The calculator generates this patch. The operator
145+
itself is configured through Helm at `operator.resources`. If Helm or GitOps
146+
owns the Config, update that source of truth instead of applying a direct patch.
147+
148+
## Validate in your environment
149+
150+
The tested profiles and calculator results are starting points, not guarantees.
151+
Cluster administrators must validate them against a representative workload
152+
lifecycle before relying on them in production. Every deployment has different
153+
workload shapes, plugins, placement constraints, and submission patterns.
154+
155+
This is deployment validation performed by the platform operator; it does not
156+
require running KAI's development scale-test suite.
157+
158+
Run cluster fill, the largest jobs, a submission burst, reclaim or preemption,
159+
and cleanup. Monitor:
160+
161+
- container working set, RSS, and Go heap;
162+
- CPU usage and throttling;
163+
- restarts and termination reasons;
164+
- scheduling latency and controller workqueue depth;
165+
- Pods, Jobs, PodGroups, BindRequests, queues, and storage/DRA objects.
166+
167+
Useful PromQL:
168+
169+
```promql
170+
max_over_time(container_memory_working_set_bytes{namespace="kai-scheduler",container!="POD"}[24h])
171+
172+
quantile_over_time(0.95, container_memory_working_set_bytes{namespace="kai-scheduler",container!="POD"}[24h])
173+
174+
sum by (container) (rate(container_cpu_usage_seconds_total{namespace="kai-scheduler",container!="POD"}[5m]))
175+
176+
max by (name, pod) (workqueue_depth{namespace="kai-scheduler"})
177+
```
178+
179+
| Symptom | Action |
180+
| --- | --- |
181+
| Memory rises during a large job | Increase scheduler limit or reduce eligible nodes with scheduler sharding. |
182+
| Memory rises during submission bursts | Increase scheduler memory or reduce simultaneous submissions. |
183+
| Memory remains high after cleanup | Wait for active scheduling and garbage collection; use complete-cycle peaks. |
184+
| Binder memory and BindRequest depth rise together | Increase Binder resources and investigate binding throughput. |
185+
| Workqueue grows while CPU is throttled | Raise CPU request and remove or increase CPU limit. |
186+
187+
Prefer no CPU limit for latency-sensitive controllers. If policy requires one,
188+
measure demand without throttling first. Repeat validation after changing KAI,
189+
Kubernetes, shards, plugins, storage, DRA, GPU sharing, or workload shape.
190+
191+
The [scale-test guide](../developer/scale-tests.md) provides examples for
192+
building representative workload tests.
193+
194+
## Vertical Pod Autoscaler
195+
196+
KAI can create VPA policies when VPA is installed. Start with `updateMode: Off`
197+
and observe recommendations through a complete workload cycle.
198+
199+
The default VPA maximum of 2 CPUs and 5 GiB is too low for the 1000-node
200+
scheduler profile. Raise the service-specific bounds before enabling VPA for
201+
that profile.
202+
203+
For the Scheduler, prefer `controlledValues: RequestsOnly` when keeping a
204+
validated static limit. If VPA lowers a scheduler Pod's memory limit in place,
205+
the cgroup limit can fall before the process has released memory. KAI updates
206+
the Go memory target after observing the cgroup change, but the Pod can be
207+
OOM-killed before garbage collection releases enough memory. Reduce the
208+
scheduler limit only after complete-cycle measurements show that it is safe.
209+
210+
See the upstream [VPA mode documentation](https://github.com/kubernetes/autoscaler/blob/master/vertical-pod-autoscaler/docs/quickstart.md)
211+
and [Scheduling shards](./scheduling-shards.md#go-memory-limit).
212+
213+
## Optional operands
214+
215+
Optional operands were not included in the profiles:
216+
217+
- Size node scale adjuster from total Pods and peak Pod-update rate.
218+
- NUMA placement exporter runs once per selected node; multiply its resources
219+
by selected node count.
220+
- Reserve capacity for the maximum simultaneous resource-reservation Pods.

0 commit comments

Comments
 (0)