Scheduler causes uneven CPU overcommit on heterogeneous Proxmox clusters

[emanuelebosetti]

Problem

On Proxmox clusters where nodes have different CPU core counts, the current round-robin scheduler distributes VMs evenly by count, ignoring CPU capacity. This causes severe CPU overcommit on nodes with fewer cores while nodes with more cores remain underutilized.

The scheduler only considers memory (schedulerHints.memoryAdjustment). CPU is never evaluated.

Real-world example

A 20-node production Proxmox cluster hosting multiple CAPI-managed Kubernetes clusters (229 VMs total across different clusters) with 3 node classes:

Class	Nodes	Cores	RAM
A	10	64	504 GB
B	2	32	661 GB
C	8	32	504 GB

Current state:

Node class	Avg CPU ratio	Max CPU ratio
A (64 cores)	2.67:1	3.25:1
B (32 cores, 661 GB RAM)	2.28:1	2.81:1
C (32 cores, 504 GB RAM)	1.41:1	1.97:1

After a rollout restart (all 229 VMs re-scheduled from zero):

Metric	Result
CPU ratio max	4.44:1
CPU ratio min	1.38:1
Spread (max-min)	3.06
Nodes above 3:1	9 / 20

Class B nodes (32 cores, highest RAM) are hit the hardest: the scheduler prefers them because they have the most available memory, but they have half the CPU cores. The imbalance persists until manual rebalancing via Proxmox live migration.

Root cause

In vmscheduler.go, selectNode:

1. Sorts nodes by available memory (descending)
2. Counts existing VMs per node
3. Picks the node with the fewest VMs that has enough memory

CPU capacity (node.CPUInfo.CPUs) and CPU allocation per VM (vm.CPUs) are available from the Proxmox API but never queried by the scheduler.

Suggestion

Introduce a schedulerHints.cpuAdjustment field (analogous to memoryAdjustment) that, when set, enables a scheduling mode that considers both CPU and memory saturation when placing VMs. When disabled (default 0), the current round-robin behavior is preserved.

We'd be happy to contribute a PR for this. Before starting, we'd like to know if this kind of scheduler improvement aligns with the project's roadmap and if there are any design preferences we should follow.

[65278]

Whilst I'm not opposed to a better scheduler, do consider that Kubernetes is supposed to be a high-availability installation and that running out of memory is a far worse condition than overbooking CPUs.

As such, this is low priority, but contributions are welcome.

As a side note, we require all LLM contributions to be marked as such.

[wikkyk]

The hypothetical cpuAdjustment and the scheduler taking CPU utilisation into account are separate issues that are tangential.

First, we would need to change the scheduling algorithm to take CPU into account. Only then it would make sense to add cpuAdjustment. However, that alone wouldn't solve your issue, which is what is more important. cpuAdjustment alone wouldn't help you here because it's merely a factor applied to core count.

A good way to solve this would be to introduce the idea of badness+tolerance (à la TeX or like swappiness in Linux): a setting for controlling what is more acceptable to be booked. This can be naturally expanded to other bookable resources.

Example:
Two machines with 8 cores and 64 GB each. Let's ignore adjustment for now. Machine 1 has 6 cores and 16 GB occupied, Machine 2 has 2 cores and 48 GB occupied. You want to deploy a machine with 2 cores and 16 GB. Neither machine is a good fit because both machines will have some resource fully booked.
If you prefer to book memory, you set memory tolerance (not adjustment, that's just a factor) higher than CPU tolerance and you deploy to Machine 2. If you prefer to book CPU, you set CPU tolerance higher and you deploy to Machine 1.

Implementation: A free core/gigabyte is worth a point. A free core/gigabyte in the overbooked (adjusted) set is worth half a point*. The score for each resource is multiplied by its tolerance factor. Then the scores are summed. The machine with the highest total score that has the available resources is the provisioning target. This will approximate the current algorithm (a machine's score is reduced as it loses free memory so machines with more free memory bubble up) given 0 CPU tolerance (=CPU scheduling disabled).
Alternative implementation: the per-core/gigabyte score is settable and we don't have the badness/tolerance setting. If you care about CPU more than memory, you set its contribution higher.

*or a function of the amount of available overbookable capacity so that overbookable capacity has a diminishing contribution; just dipping your toes into the overbookable capacity, while bad, is a lot better than being almost out of overbookable capacity,

[emanuelebosetti]

We had a simpler approach in mind: a bottleneck-saturation model where for each candidate node, after hypothetical placement, we compute:

• cpuSat = (allocatedCPU + requestedCPU) / allocatableCPU
• memSat = (allocatedMem + requestedMem) / allocatableMem
• bottleneck = max(cpuSat, memSat)

Then pick the node with the lowest bottleneck — "find the node that's least stressed on its tightest resource." Nodes that can't fit the VM on either dimension are excluded upfront as hard constraints.

cpuAdjustment was pulling double duty as both the on/off switch for CPU awareness and the overcommit ratio — we were aware that's not great but it kept the API surface minimal for a first pass.

Your point about separating adjustment from tolerance is well taken — our model can't express "I'd rather saturate CPU than memory" which is exactly the kind of knob you'd want in practice.
We already have the plumbing in place (client API for GetReservableCPUCores, error types, test harness), so refactoring the scoring function to a weighted-sum model shouldn't be a huge lift.