Skip to content

feat(scheduler): account for NvFractions GPU memory - #2041

Draft
davidLif wants to merge 4 commits into
codex/nvfractions-foundationfrom
codex/nvfractions-scheduler-accounting
Draft

feat(scheduler): account for NvFractions GPU memory#2041
davidLif wants to merge 4 commits into
codex/nvfractions-foundationfrom
codex/nvfractions-scheduler-accounting

Conversation

@davidLif

@davidLif davidLif commented Aug 10, 2026

Copy link
Copy Markdown
Collaborator

Description

Stack PR 2/9 for GPU sharing migration.

Base: codex/nvfractions-foundation
Head: codex/nvfractions-scheduler-accounting

This wires NvFractions GPU memory requests into scheduler-side resource accounting. It updates pod and node resource calculations, pod group/resource controller accounting, node scale adjustment calculations, and related fake builders/tests so NvFractions requests are counted consistently with the common parser introduced in PR 1.

Related Issues

N/A

Checklist

Note: Ensure your PR title follows the Conventional Commits format (e.g., feat(scheduler): add new feature)

  • Self-reviewed
  • Added/updated tests (if needed)
  • Updated documentation (if needed)
  • Added a changelog fragment via make changelog (or applied the skip-changelog label). Do not edit CHANGELOG.md directly — pending fragments are folded into it at release time.

Breaking Changes

None.

Additional Notes

Validation run:

go test ./pkg/nodescaleadjuster/scale_adjuster ./pkg/podgroupcontroller/controllers/resources ./pkg/scheduler/api/node_info ./pkg/scheduler/api/pod_info

@davidLif
davidLif force-pushed the codex/nvfractions-scheduler-accounting branch from 7afd5cd to 6b54e5f Compare August 10, 2026 10:02
@davidLif
davidLif force-pushed the codex/nvfractions-scheduler-accounting branch from 6b54e5f to 23afd8f Compare August 11, 2026 14:03
@github-actions

github-actions Bot commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown

📊 Performance Benchmark Results

Comparing PR (codex/nvfractions-scheduler-accounting) vs main branch — click to expand 
goos: linux
goarch: amd64
pkg: github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/actions
cpu: AMD EPYC 7763 64-Core Processor                
                                    │ main-bench.txt │           pr-bench.txt            │
                                    │     sec/op     │   sec/op     vs base              │
AllocateAction_SmallCluster-4            107.5m ± 0%   108.3m ± 7%  +0.69% (p=0.002 n=6)
AllocateAction_MediumCluster-4           133.0m ± 1%   135.0m ± 1%  +1.51% (p=0.004 n=6)
AllocateAction_LargeCluster-4            195.1m ± 2%   197.2m ± 3%       ~ (p=0.093 n=6)
ReclaimAction_SmallCluster-4             103.1m ± 0%   103.5m ± 0%  +0.38% (p=0.004 n=6)
ReclaimAction_MediumCluster-4            106.2m ± 1%   106.6m ± 1%  +0.43% (p=0.015 n=6)
PreemptAction_SmallCluster-4             104.3m ± 0%   104.6m ± 0%  +0.32% (p=0.004 n=6)
PreemptAction_MediumCluster-4            114.8m ± 1%   115.6m ± 1%  +0.73% (p=0.041 n=6)
ConsolidationAction_SmallCluster-4       122.0m ± 1%   122.6m ± 1%  +0.48% (p=0.026 n=6)
ConsolidationAction_MediumCluster-4      261.0m ± 1%   264.3m ± 1%  +1.25% (p=0.002 n=6)
FullSchedulingCycle_SmallCluster-4       105.6m ± 0%   106.2m ± 0%  +0.59% (p=0.002 n=6)
FullSchedulingCycle_MediumCluster-4      119.0m ± 2%   120.1m ± 1%       ~ (p=0.240 n=6)
FullSchedulingCycle_LargeCluster-4       154.5m ± 1%   155.6m ± 1%       ~ (p=0.180 n=6)
ManyQueues_MediumCluster-4               135.7m ± 1%   136.8m ± 1%       ~ (p=0.093 n=6)
GangScheduling_MediumCluster-4           153.8m ± 2%   155.7m ± 2%       ~ (p=0.394 n=6)
geomean                                  131.7m        132.7m       +0.80%

                                    │ main-bench.txt │            pr-bench.txt            │
                                    │      B/op      │     B/op      vs base              │
AllocateAction_SmallCluster-4           2.084Mi ± 1%   2.080Mi ± 0%       ~ (p=0.485 n=6)
AllocateAction_MediumCluster-4          9.903Mi ± 0%   9.909Mi ± 0%  +0.06% (p=0.002 n=6)
AllocateAction_LargeCluster-4           31.26Mi ± 0%   31.27Mi ± 0%       ~ (p=0.240 n=6)
ReclaimAction_SmallCluster-4            952.2Ki ± 1%   956.0Ki ± 0%       ~ (p=0.180 n=6)
ReclaimAction_MediumCluster-4           3.103Mi ± 0%   3.106Mi ± 0%       ~ (p=0.065 n=6)
PreemptAction_SmallCluster-4            1.346Mi ± 0%   1.348Mi ± 1%       ~ (p=0.065 n=6)
PreemptAction_MediumCluster-4           5.936Mi ± 0%   5.941Mi ± 0%  +0.08% (p=0.041 n=6)
ConsolidationAction_SmallCluster-4      8.208Mi ± 0%   8.207Mi ± 0%       ~ (p=0.937 n=6)
ConsolidationAction_MediumCluster-4     67.76Mi ± 0%   67.77Mi ± 0%       ~ (p=0.132 n=6)
FullSchedulingCycle_SmallCluster-4      1.409Mi ± 1%   1.406Mi ± 1%       ~ (p=0.818 n=6)
FullSchedulingCycle_MediumCluster-4     6.140Mi ± 0%   6.146Mi ± 0%  +0.09% (p=0.002 n=6)
FullSchedulingCycle_LargeCluster-4      18.35Mi ± 0%   18.36Mi ± 0%  +0.06% (p=0.026 n=6)
ManyQueues_MediumCluster-4              13.13Mi ± 0%   13.14Mi ± 0%  +0.02% (p=0.009 n=6)
GangScheduling_MediumCluster-4          15.33Mi ± 0%   15.34Mi ± 0%  +0.08% (p=0.002 n=6)
geomean                                 6.545Mi        6.547Mi       +0.04%

                                    │ main-bench.txt │           pr-bench.txt            │
                                    │   allocs/op    │  allocs/op   vs base              │
AllocateAction_SmallCluster-4            31.49k ± 0%   31.59k ± 0%  +0.31% (p=0.002 n=6)
AllocateAction_MediumCluster-4           245.8k ± 0%   246.2k ± 0%  +0.16% (p=0.002 n=6)
AllocateAction_LargeCluster-4            1.007M ± 0%   1.008M ± 0%  +0.10% (p=0.002 n=6)
ReclaimAction_SmallCluster-4             8.937k ± 0%   8.988k ± 0%  +0.57% (p=0.002 n=6)
ReclaimAction_MediumCluster-4            28.12k ± 0%   28.32k ± 0%  +0.71% (p=0.002 n=6)
PreemptAction_SmallCluster-4             15.27k ± 0%   15.34k ± 0%  +0.50% (p=0.002 n=6)
PreemptAction_MediumCluster-4            55.29k ± 0%   55.59k ± 0%  +0.54% (p=0.002 n=6)
ConsolidationAction_SmallCluster-4       109.7k ± 0%   109.7k ± 0%  +0.07% (p=0.002 n=6)
ConsolidationAction_MediumCluster-4      1.020M ± 0%   1.021M ± 0%  +0.03% (p=0.002 n=6)
FullSchedulingCycle_SmallCluster-4       19.63k ± 0%   19.70k ± 0%  +0.36% (p=0.002 n=6)
FullSchedulingCycle_MediumCluster-4      136.3k ± 0%   136.6k ± 0%  +0.22% (p=0.002 n=6)
FullSchedulingCycle_LargeCluster-4       537.7k ± 0%   538.4k ± 0%  +0.14% (p=0.002 n=6)
ManyQueues_MediumCluster-4               274.1k ± 0%   274.4k ± 0%  +0.14% (p=0.002 n=6)
GangScheduling_MediumCluster-4           455.3k ± 0%   456.1k ± 0%  +0.18% (p=0.002 n=6)
geomean                                  109.0k        109.3k       +0.29%

pkg: github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/actions/integration_tests/reclaim
                                                          │ main-bench.txt │           pr-bench.txt            │
                                                          │     sec/op     │   sec/op     vs base              │
ReclaimLargeJobs_10Node-4                                      134.9m ± 1%   136.7m ± 1%  +1.38% (p=0.004 n=6)
ReclaimLargeJobs_50Node-4                                      312.0m ± 1%   320.0m ± 3%  +2.58% (p=0.002 n=6)
ReclaimLargeJobs_100Node-4                                     626.8m ± 9%   648.0m ± 6%  +3.38% (p=0.041 n=6)
ReclaimLargeJobs_200Node-4                                      1.576 ± 1%    1.614 ± 1%  +2.45% (p=0.002 n=6)
ReclaimLargeJobs_500Node-4                                      7.316 ± 1%    7.328 ± 2%       ~ (p=0.310 n=6)
ReclaimLargeJobs_1000Node-4                                     32.41 ± 2%    32.25 ± 1%       ~ (p=0.699 n=6)
ReclaimManySingleGPUJobsFullCycle_10Node-4                     156.4m ± 1%   158.9m ± 1%  +1.60% (p=0.009 n=6)
ReclaimManySingleGPUJobsFullCycle_50Node-4                     961.3m ± 1%   970.2m ± 1%  +0.92% (p=0.009 n=6)
ReclaimManySingleGPUJobsFullCycle_100Node-4                     3.338 ± 0%    3.369 ± 1%  +0.93% (p=0.002 n=6)
ReclaimManySingleGPUJobsFullCycle_200Node-4                     13.22 ± 1%    13.39 ± 7%  +1.33% (p=0.002 n=6)
ReclaimManySingleGPUJobsFullCycle_500Node-4                     89.66 ± 3%    94.35 ± 7%       ~ (p=0.093 n=6)
ReclaimManySingleGPUJobsFullCycleWithMinRuntime_500Node-4       88.42 ± 6%    87.62 ± 5%       ~ (p=0.937 n=6)
geomean                                                         2.994         3.040       +1.53%

                                                          │ main-bench.txt │            pr-bench.txt            │
                                                          │      B/op      │     B/op      vs base              │
ReclaimLargeJobs_10Node-4                                     17.22Mi ± 1%   17.20Mi ± 1%       ~ (p=0.394 n=6)
ReclaimLargeJobs_50Node-4                                     91.88Mi ± 0%   91.89Mi ± 0%       ~ (p=0.699 n=6)
ReclaimLargeJobs_100Node-4                                    204.3Mi ± 0%   204.3Mi ± 0%       ~ (p=0.818 n=6)
ReclaimLargeJobs_200Node-4                                    486.2Mi ± 0%   486.3Mi ± 0%       ~ (p=0.240 n=6)
ReclaimLargeJobs_500Node-4                                    1.698Gi ± 0%   1.698Gi ± 0%       ~ (p=1.000 n=6)
ReclaimLargeJobs_1000Node-4                                   5.141Gi ± 0%   5.141Gi ± 0%       ~ (p=0.310 n=6)
ReclaimManySingleGPUJobsFullCycle_10Node-4                    14.80Mi ± 0%   14.81Mi ± 0%       ~ (p=0.937 n=6)
ReclaimManySingleGPUJobsFullCycle_50Node-4                    145.0Mi ± 0%   145.0Mi ± 0%       ~ (p=0.065 n=6)
ReclaimManySingleGPUJobsFullCycle_100Node-4                   470.6Mi ± 0%   470.6Mi ± 0%       ~ (p=0.589 n=6)
ReclaimManySingleGPUJobsFullCycle_200Node-4                   1.630Gi ± 0%   1.630Gi ± 0%       ~ (p=0.818 n=6)
ReclaimManySingleGPUJobsFullCycle_500Node-4                   10.35Gi ± 0%   10.35Gi ± 0%       ~ (p=0.589 n=6)
ReclaimManySingleGPUJobsFullCycleWithMinRuntime_500Node-4     10.35Gi ± 0%   10.35Gi ± 0%       ~ (p=0.065 n=6)
geomean                                                       504.6Mi        504.5Mi       -0.00%

                                                          │ main-bench.txt │           pr-bench.txt            │
                                                          │   allocs/op    │  allocs/op   vs base              │
ReclaimLargeJobs_10Node-4                                      242.0k ± 0%   242.2k ± 0%  +0.05% (p=0.037 n=6)
ReclaimLargeJobs_50Node-4                                      1.482M ± 0%   1.482M ± 0%  +0.05% (p=0.002 n=6)
ReclaimLargeJobs_100Node-4                                     3.741M ± 0%   3.742M ± 0%  +0.03% (p=0.002 n=6)
ReclaimLargeJobs_200Node-4                                     10.52M ± 0%   10.52M ± 0%  +0.03% (p=0.002 n=6)
ReclaimLargeJobs_500Node-4                                     45.58M ± 0%   45.58M ± 0%  +0.01% (p=0.002 n=6)
ReclaimLargeJobs_1000Node-4                                    159.5M ± 0%   159.5M ± 0%  +0.01% (p=0.002 n=6)
ReclaimManySingleGPUJobsFullCycle_10Node-4                     279.9k ± 0%   280.3k ± 0%  +0.11% (p=0.041 n=6)
ReclaimManySingleGPUJobsFullCycle_50Node-4                     3.742M ± 0%   3.743M ± 0%  +0.04% (p=0.002 n=6)
ReclaimManySingleGPUJobsFullCycle_100Node-4                    13.33M ± 0%   13.34M ± 0%  +0.02% (p=0.002 n=6)
ReclaimManySingleGPUJobsFullCycle_200Node-4                    50.07M ± 0%   50.07M ± 0%  +0.01% (p=0.002 n=6)
ReclaimManySingleGPUJobsFullCycle_500Node-4                    300.6M ± 0%   300.6M ± 0%  +0.01% (p=0.002 n=6)
ReclaimManySingleGPUJobsFullCycleWithMinRuntime_500Node-4      300.6M ± 0%   300.6M ± 0%  +0.01% (p=0.002 n=6)
geomean                                                        11.69M        11.69M       +0.03%

                                                          │ main-bench.txt │              pr-bench.txt              │
                                                          │ full_cycles/op │ full_cycles/op  vs base                │
ReclaimManySingleGPUJobsFullCycle_10Node-4                      1.000 ± 0%       1.000 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimManySingleGPUJobsFullCycle_50Node-4                      1.000 ± 0%       1.000 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimManySingleGPUJobsFullCycle_100Node-4                     1.000 ± 0%       1.000 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimManySingleGPUJobsFullCycle_200Node-4                     1.000 ± 0%       1.000 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimManySingleGPUJobsFullCycle_500Node-4                     1.000 ± 0%       1.000 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimManySingleGPUJobsFullCycleWithMinRuntime_500Node-4       1.000 ± 0%       1.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                                         1.000            1.000       +0.00%
¹ all samples are equal

                                            │          main-bench.txt           │                     pr-bench.txt                     │
                                            │ fit_error_tasks_after_allocate/op │ fit_error_tasks_after_allocate/op  vs base           │
ReclaimManySingleGPUJobsFullCycle_500Node-4                         4.000k ± 0%                         4.000k ± 0%  ~ (p=1.000 n=6) ¹
¹ all samples are equal

                                            │          main-bench.txt           │                    pr-bench.txt                    │
                                            │ heap_live_after_allocate_bytes/op │ heap_live_after_allocate_bytes/op  vs base         │
ReclaimManySingleGPUJobsFullCycle_500Node-4                        41.55M ± 76%                        41.55M ± 76%  ~ (p=0.699 n=6)

                                            │         main-bench.txt         │                  pr-bench.txt                   │
                                            │ heap_live_after_cycle_bytes/op │ heap_live_after_cycle_bytes/op  vs base         │
ReclaimManySingleGPUJobsFullCycle_500Node-4                     56.15M ± 56%                     56.16M ± 56%  ~ (p=0.937 n=6)

pkg: github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/actions/preempt
                                  │ main-bench.txt │         pr-bench.txt         │
                                  │     sec/op     │   sec/op     vs base         │
OrderedVictimsQueueConstruction-4      727.1µ ± 3%   716.2µ ± 5%  ~ (p=0.589 n=6)

                                  │ main-bench.txt │          pr-bench.txt           │
                                  │      B/op      │     B/op      vs base           │
OrderedVictimsQueueConstruction-4     114.9Ki ± 0%   114.9Ki ± 0%  ~ (p=1.000 n=6) ¹
¹ all samples are equal

                                  │ main-bench.txt │          pr-bench.txt          │
                                  │   allocs/op    │  allocs/op   vs base           │
OrderedVictimsQueueConstruction-4      4.033k ± 0%   4.033k ± 0%  ~ (p=1.000 n=6) ¹
¹ all samples are equal

pkg: github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/actions/reclaim
                                                         │ main-bench.txt │           pr-bench.txt            │
                                                         │     sec/op     │   sec/op     vs base              │
ReclaimUnschedulableDistributedJob_10Node-4                   145.5m ± 1%   145.7m ± 1%       ~ (p=0.394 n=6)
ReclaimUnschedulableDistributedJob_50Node-4                    3.403 ± 1%    3.404 ± 4%       ~ (p=0.937 n=6)
ReclaimUnschedulableDistributedJob_100Node-4                   15.99 ± 1%    16.04 ± 1%       ~ (p=0.394 n=6)
ReclaimUnschedulableDistributedJob_AntiAffinity100Node-4       14.71 ± 0%    14.72 ± 1%       ~ (p=0.818 n=6)
ReclaimWithMissingPVCJobs-4                                   2.500m ± 4%   2.427m ± 8%       ~ (p=0.132 n=6)
geomean                                                       781.3m        777.6m       -0.47%

                                                         │ main-bench.txt │            pr-bench.txt            │
                                                         │      B/op      │     B/op      vs base              │
ReclaimUnschedulableDistributedJob_10Node-4                  20.91Mi ± 3%   20.92Mi ± 3%       ~ (p=0.589 n=6)
ReclaimUnschedulableDistributedJob_50Node-4                  1.155Gi ± 0%   1.154Gi ± 0%       ~ (p=0.485 n=6)
ReclaimUnschedulableDistributedJob_100Node-4                 5.560Gi ± 0%   5.560Gi ± 0%       ~ (p=0.699 n=6)
ReclaimUnschedulableDistributedJob_AntiAffinity100Node-4     5.159Gi ± 0%   5.159Gi ± 0%       ~ (p=0.699 n=6)
ReclaimWithMissingPVCJobs-4                                  8.992Ki ± 1%   9.086Ki ± 8%       ~ (p=0.364 n=6)
geomean                                                      91.83Mi        92.02Mi       +0.21%

                                                         │ main-bench.txt │           pr-bench.txt            │
                                                         │   allocs/op    │  allocs/op   vs base              │
ReclaimUnschedulableDistributedJob_10Node-4                   306.1k ± 3%   306.3k ± 3%  +0.05% (p=0.041 n=6)
ReclaimUnschedulableDistributedJob_50Node-4                   27.77M ± 0%   27.77M ± 0%       ~ (p=0.394 n=6)
ReclaimUnschedulableDistributedJob_100Node-4                  169.0M ± 0%   169.0M ± 0%       ~ (p=0.818 n=6)
ReclaimUnschedulableDistributedJob_AntiAffinity100Node-4      146.8M ± 0%   146.8M ± 0%       ~ (p=0.937 n=6)
ReclaimWithMissingPVCJobs-4                                    169.5 ± 0%    170.0 ± 3%       ~ (p=0.364 n=6)
geomean                                                       2.045M        2.046M       +0.07%

                                                         │ main-bench.txt │             pr-bench.txt             │
                                                         │  duplicate/op  │ duplicate/op  vs base                │
ReclaimUnschedulableDistributedJob_10Node-4                  0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimUnschedulableDistributedJob_50Node-4                  366.0 ± 0%       366.0 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimUnschedulableDistributedJob_100Node-4                1.020k ± 0%      1.020k ± 0%       ~ (p=1.000 n=6) ¹
ReclaimUnschedulableDistributedJob_AntiAffinity100Node-4    1.020k ± 0%      1.020k ± 0%       ~ (p=1.000 n=6) ¹
geomean                                                                 ²                 +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                                         │ main-bench.txt │             pr-bench.txt             │
                                                         │  simulated/op  │ simulated/op  vs base                │
ReclaimUnschedulableDistributedJob_10Node-4                    18.00 ± 0%     18.00 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimUnschedulableDistributedJob_50Node-4                    178.0 ± 0%     178.0 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimUnschedulableDistributedJob_100Node-4                   378.0 ± 0%     378.0 ± 0%       ~ (p=1.000 n=6) ¹
ReclaimUnschedulableDistributedJob_AntiAffinity100Node-4       378.0 ± 0%     378.0 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                                        146.3          146.3       +0.00%
¹ all samples are equal

Legend

  • 📉 Negative delta = Performance improvement (faster)
  • 📈 Positive delta = Performance regression (slower)
  • p-value < 0.05 indicates statistically significant change
Raw benchmark data

PR branch:

goos: linux
goarch: amd64
pkg: github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/actions
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkAllocateAction_SmallCluster-4         	       9	 115472919 ns/op	 2188070 B/op	   31590 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 108334477 ns/op	 2173281 B/op	   31586 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 108225616 ns/op	 2181552 B/op	   31593 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 108600131 ns/op	 2172644 B/op	   31584 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 108026742 ns/op	 2185428 B/op	   31588 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 108170021 ns/op	 2180223 B/op	   31579 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 135271103 ns/op	10388715 B/op	  246158 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 135096931 ns/op	10391556 B/op	  246125 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 135120405 ns/op	10390207 B/op	  246180 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 134689015 ns/op	10389284 B/op	  246170 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 134122114 ns/op	10390645 B/op	  246192 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 134943899 ns/op	10389808 B/op	  246180 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 198587286 ns/op	32802541 B/op	 1008224 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 198057287 ns/op	32789417 B/op	 1008138 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 196323991 ns/op	32786105 B/op	 1008171 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 195236255 ns/op	32789402 B/op	 1008209 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 191939402 ns/op	32790098 B/op	 1008199 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 198979697 ns/op	32816100 B/op	 1008153 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103551082 ns/op	  978554 B/op	    8986 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103983515 ns/op	  975392 B/op	    8987 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103340820 ns/op	  979250 B/op	    8988 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103543414 ns/op	  974941 B/op	    8984 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103507445 ns/op	  979234 B/op	    8988 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103248605 ns/op	  979260 B/op	    8988 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 106904606 ns/op	 3257475 B/op	   28318 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 107850079 ns/op	 3256510 B/op	   28320 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 106709909 ns/op	 3256294 B/op	   28319 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 106556040 ns/op	 3256146 B/op	   28318 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 106544088 ns/op	 3260612 B/op	   28320 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 106399260 ns/op	 3260117 B/op	   28320 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104552629 ns/op	 1413474 B/op	   15344 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104642071 ns/op	 1413693 B/op	   15345 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104460784 ns/op	 1413293 B/op	   15345 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104649881 ns/op	 1405332 B/op	   15342 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104613694 ns/op	 1413213 B/op	   15345 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104727210 ns/op	 1419511 B/op	   15343 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 116473561 ns/op	 6233001 B/op	   55593 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 115355957 ns/op	 6224955 B/op	   55592 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 116467683 ns/op	 6220880 B/op	   55593 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 115550794 ns/op	 6241492 B/op	   55595 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 115698126 ns/op	 6228985 B/op	   55593 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 115349808 ns/op	 6229672 B/op	   55596 allocs/op
BenchmarkConsolidationAction_SmallCluster-4    	       9	 122739404 ns/op	 8602268 B/op	  109725 allocs/op
BenchmarkConsolidationAction_SmallCluster-4    	       9	 122804265 ns/op	 8602613 B/op	  109735 allocs/op
BenchmarkConsolidationAction_SmallCluster-4    	       9	 122730879 ns/op	 8606142 B/op	  109760 allocs/op
BenchmarkConsolidationAction_SmallCluster-4    	       9	 122413960 ns/op	 8616821 B/op	  109748 allocs/op

Main branch:

goos: linux
goarch: amd64
pkg: github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/actions
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkAllocateAction_SmallCluster-4         	      10	 107616820 ns/op	 2204153 B/op	   31493 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 107313432 ns/op	 2172456 B/op	   31492 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 107632677 ns/op	 2188181 B/op	   31490 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 107830614 ns/op	 2175000 B/op	   31489 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 107363621 ns/op	 2182904 B/op	   31482 allocs/op
BenchmarkAllocateAction_SmallCluster-4         	      10	 107462340 ns/op	 2187129 B/op	   31480 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 132594015 ns/op	10382223 B/op	  245793 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 132385799 ns/op	10384302 B/op	  245776 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 134584184 ns/op	10385704 B/op	  245774 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 133502302 ns/op	10383288 B/op	  245787 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 133440267 ns/op	10383678 B/op	  245782 allocs/op
BenchmarkAllocateAction_MediumCluster-4        	       8	 132457679 ns/op	10383949 B/op	  245795 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 195424963 ns/op	32773162 B/op	 1007140 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 193170032 ns/op	32774534 B/op	 1007175 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 191389676 ns/op	32778826 B/op	 1007164 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 195661129 ns/op	32775357 B/op	 1007213 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 195048152 ns/op	32800304 B/op	 1007161 allocs/op
BenchmarkAllocateAction_LargeCluster-4         	       6	 195092406 ns/op	32820178 B/op	 1007161 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103214862 ns/op	  977647 B/op	    8935 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103318079 ns/op	  981634 B/op	    8938 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103056089 ns/op	  974916 B/op	    8938 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 102852053 ns/op	  974491 B/op	    8936 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 103235236 ns/op	  974287 B/op	    8936 allocs/op
BenchmarkReclaimAction_SmallCluster-4          	      10	 102981580 ns/op	  975100 B/op	    8938 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 106704488 ns/op	 3252904 B/op	   28119 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 106322233 ns/op	 3253740 B/op	   28119 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 106382465 ns/op	 3257453 B/op	   28120 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 105827405 ns/op	 3249364 B/op	   28118 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 105897657 ns/op	 3253084 B/op	   28119 allocs/op
BenchmarkReclaimAction_MediumCluster-4         	      10	 106022329 ns/op	 3256904 B/op	   28120 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104292727 ns/op	 1412548 B/op	   15271 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104470849 ns/op	 1408007 B/op	   15267 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104414691 ns/op	 1408143 B/op	   15268 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104301189 ns/op	 1411956 B/op	   15270 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104177742 ns/op	 1411932 B/op	   15269 allocs/op
BenchmarkPreemptAction_SmallCluster-4          	      10	 104212949 ns/op	 1411658 B/op	   15268 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 115508411 ns/op	 6224861 B/op	   55294 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 115599991 ns/op	 6228614 B/op	   55296 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 114836114 ns/op	 6224527 B/op	   55294 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 114730698 ns/op	 6224552 B/op	   55293 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 114356980 ns/op	 6215798 B/op	   55291 allocs/op
BenchmarkPreemptAction_MediumCluster-4         	       9	 114087637 ns/op	 6219766 B/op	   55291 allocs/op
BenchmarkConsolidationAction_SmallCluster-4    	       9	 121917044 ns/op	 8616571 B/op	  109642 allocs/op
BenchmarkConsolidationAction_SmallCluster-4    	       9	 121392479 ns/op	 8610168 B/op	  109657 allocs/op
BenchmarkConsolidationAction_SmallCluster-4    	       9	 121253132 ns/op	 8602776 B/op	  109670 allocs/op
BenchmarkConsolidationAction_SmallCluster-4    	       9	 122081643 ns/op	 8599555 B/op	  109653 allocs/op

@davidLif
davidLif force-pushed the codex/nvfractions-scheduler-accounting branch from e33946a to b2db18b Compare August 11, 2026 21:23
@davidLif
davidLif force-pushed the codex/nvfractions-scheduler-accounting branch from b2db18b to 32e3d88 Compare August 13, 2026 11:25
@github-actions

github-actions Bot commented Aug 13, 2026

Copy link
Copy Markdown

Total coverage: 54.6% -> 54.7% (delta 0.10%)

Merging this branch changes the coverage (4 decrease, 1 increase)

Impacted Packages Coverage Δ 🤖
github.com/kai-scheduler/KAI-scheduler/pkg/binder/binding/resourcereservation 91.41% (-1.39%) 👎
github.com/kai-scheduler/KAI-scheduler/pkg/nodescaleadjuster/scale_adjuster 97.60% (-0.81%) 👎
github.com/kai-scheduler/KAI-scheduler/pkg/podgroupcontroller/controllers 43.75% (ø)
github.com/kai-scheduler/KAI-scheduler/pkg/podgroupcontroller/controllers/resources 81.61% (-1.22%) 👎
github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/actions/common/solvers/accumulated_scenario_filters/idle_gpus 86.70% (ø)
github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/node_info 78.32% (+1.29%) 👍
github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/pod_info 72.47% (-0.35%) 👎
github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/test_utils/jobs_fake 61.01% (ø)
github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/test_utils/tasks_fake 0.00% (ø)
Coverage by file

Changed files (no unit tests)

Changed File Coverage Δ Total Covered Missed 🤖
github.com/kai-scheduler/KAI-scheduler/pkg/binder/binding/resourcereservation/resource_reservation.go 91.41% (-1.39%) 256 (+20) 234 (+15) 22 (+5) 👎
github.com/kai-scheduler/KAI-scheduler/pkg/nodescaleadjuster/scale_adjuster/calculator.go 97.44% (-2.56%) 39 38 (-1) 1 (+1) 👎
github.com/kai-scheduler/KAI-scheduler/pkg/nodescaleadjuster/scale_adjuster/scale_adjuster.go 97.67% (-0.03%) 86 (-1) 84 (-1) 2 👎
github.com/kai-scheduler/KAI-scheduler/pkg/podgroupcontroller/controllers/resources/fraction.go 96.00% (-2.08%) 50 (-2) 48 (-3) 2 (+1) 👎
github.com/kai-scheduler/KAI-scheduler/pkg/podgroupcontroller/controllers/resources/requested.go 80.00% (ø) 20 (-10) 16 (-8) 4 (-2)
github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/node_info/node_info.go 75.32% (+0.32%) 312 235 (+1) 77 (-1) 👍
github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/pod_info/pod_info.go 67.89% (-0.53%) 190 129 (-1) 61 (+1) 👎
github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/test_utils/jobs_fake/jobs.go 61.01% (ø) 159 97 62
github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/test_utils/tasks_fake/tasks.go 0.00% (ø) 46 (+2) 0 46 (+2)

Please note that the "Total", "Covered", and "Missed" counts above refer to code statements instead of lines of code. The value in brackets refers to the test coverage of that file in the old version of the code.

Changed unit test files

  • github.com/kai-scheduler/KAI-scheduler/pkg/binder/binding/resourcereservation/resource_reservation_test.go
  • github.com/kai-scheduler/KAI-scheduler/pkg/podgroupcontroller/controllers/resources/fraction_test.go
  • github.com/kai-scheduler/KAI-scheduler/pkg/podgroupcontroller/controllers/resources/requested_test.go
  • github.com/kai-scheduler/KAI-scheduler/pkg/podgroupcontroller/controllers/status_updater_test.go
  • github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/actions/common/solvers/accumulated_scenario_filters/idle_gpus/idle_gpus_test.go
  • github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/node_info/node_info_benchmark_test.go
  • github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/node_info/node_info_test.go
  • github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/pod_info/pod_info_test.go

@davidLif
davidLif force-pushed the codex/nvfractions-scheduler-accounting branch from 32e3d88 to 0bb6f3b Compare August 17, 2026 06:39
@davidLif
fix(binder): preserve reservations during bind cache lag
dc21431 
Signed-off-by: davidLif <davidshani12@gmail.com>
@davidLif
feat(scheduler): read NvFractions GPU memory requests
a5ae68b 
Signed-off-by: davidLif <davidshani12@gmail.com>
@davidLif
feat(scheduler): account for NvFractions GPU memory
dd7774a 
Signed-off-by: davidLif <davidshani12@gmail.com>
@davidLif
Removed unused function
432b780 
Signed-off-by: davidLif <davidshani12@gmail.com>
@davidLif
davidLif force-pushed the codex/nvfractions-scheduler-accounting branch from 0bb6f3b to 432b780 Compare August 17, 2026 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@davidLif