High CPU usage during node consolidation at scale #2972

Description

Observed Behavior:

As part of Performance and Scale testing on OpenShift ROSA HCP clusters we observed that Karpenter CPU usage grows faster than expected with node count during consolidation. On a ~500 node cluster Karpenter sustains ~3 vCPUs mean, peaking at 4 vCPUs.

We captured continuous pprof profiles (every 30s) during steady pod creation. The profiles show that the dominant CPU cost is in SimulateScheduling, where most of the time is spent on scheduler setup that appears to be repeated per candidate evaluation.

Observed CPU usage across cluster sizes:

500 nodes:

360 nodes:

110 nodes:

60 nodes:

10 nodes:

Expected Behavior:

We expected consolidation CPU overhead to be lower. The pprof profiles suggest that much of the per-candidate scheduler setup work (daemon pod compatibility, topology domain groups, instance type filtering, deep copies) is repeated across candidate evaluations even though the results don't change between candidates.

Reproduction Steps (Please include YAML):

Our test workload deploys pods in sequential waves with a 30-second delay between waves. Each wave creates a Deployment with 111 pods, each requesting 500m CPU. The waves are spread across three Karpenter NodePools targeting different instance families. The steady ramp of identical waves produces a linear node scale-up, reaching ~500 Karpenter-managed nodes by the end.

Versions:
Karpenter version: v1.9.0
Kubernetes: Kubernetes 1.35.3(OpenShift ROSA HCP 4.22)

pprof profiles captured during peak activity on the ~500-node cluster. Profile (peak)
GC dominates at 39.5% (gcBgMarkWorker → scanobject → findObject), driven by allocation churn from repeated scheduler construction. The controller loop (Start.func1.1) accounts for 49.23% cumulative, nearly all of it flows through SimulateScheduling.

Within SimulateScheduling, 78.85% of CPU goes to NewScheduler (scheduler construction), 18.95% to DeepCopyNodes, and only 1.12% to Scheduler.Solve.

Observations

SingleNodeConsolidation.ComputeCommands() evaluates every candidate node by calling SimulateScheduling(). Each call:

• Deep-copies the entire cluster state (cluster.DeepCopyNodes()) — 18.95% of SimulateScheduling CPU
• Constructs a new scheduler (provisioner.NewScheduler()) — 78.85% of CPU, including:
  • calculateExistingNodeClaims which calls getCompatibleDaemonPods for every node, which calls isDaemonPodCompatibleWithNode for every daemon pod , each allocating fresh Requirements via NewLabelRequirements (15+ sets.Set[string] map allocations per call)
  • NewTopology which calls buildDomainGroups from scratch

The call chain seems to be:

SingleNodeConsolidation.ComputeCommands()
  for each candidate:                           // N candidates (up to all active nodes)
    → computeConsolidation()
      → SimulateScheduling()
        → cluster.DeepCopyNodes()               // copies ALL N nodes every time
        → provisioner.NewScheduler()            // full scheduler construction
          → calculateExistingNodeClaims()
              for each node:                    // N nodes
                → getCompatibleDaemonPods()
                  for each daemon pod:          // M daemons
                    → isDaemonPodCompatibleWithNode()
                      → NewLabelRequirements()  // 15+ map allocs per call
                      → NewStrictPodRequirements()
          → NewTopology()
            → buildDomainGroups()
        → scheduler.Solve()                     // ~1% of total time

With 500 nodes and 500 candidates this results in 500 full NewScheduler constructions and 500 full DeepCopyNodes calls. Each NewScheduler iterates over all nodes and daemon pods for compatibility. The most expensive operations within NewScheduler don't seem to depend on which candidate node is being removed. Would it be possible to cache some of this work across candidate evaluations within a single consolidation pass?

Please let me know if I can provide further pprof's for debug or test specific scenarios.

Thank you!