perf: Attempt to improve SimulateScheduling efficiency

[GnatorX]

Fixes #2972

Problem

We don't need to deep copy of all cluster nodes. From what I can tell we do deep copy for two reasons.

1. Keep the change simulation is making isolated from shared memory of cluster
2. Get a fresh copy a consistent view of the world during candidate eval and SimulateScheduling such that the simulation aren't getting affected by live changes

But deep copy is problematic because:

1. We do a deep copy every SimulateScheduling which contains a copy of all nodes + meta tracking information. This is both computational and memory intensive.
2. We have possible lock contention since deepcopy needs a readlock against cluster. Its possible we content against updates on nodes and vice versa.

Memory Isolation

Looking at the code, we don't need to do the isolation because we only update state nodes during simulation for 2 fields, HostPortUsage and VolumeUsage. We will make this part of statenode and avoid deep copying actual node object.

We don't mutate underlying nodes too much (most happen against statenode) so attempting to move all nodes data that gets mutated to statenode so we can share cluster.nodes() data.

When controller.disrupt() happens we make a deep copy:

karpenter/pkg/controllers/disruption/helpers.go

Line 207 in d1aacae

allNodes := cluster.DeepCopyNodes()

Then each disruption method (consolidation and drift) runs SimulateScheduling

karpenter/pkg/controllers/disruption/helpers.go

Lines 53 to 57 in d1aacae

	func SimulateScheduling(ctx context.Context, kubeClient client.Client, cluster *state.Cluster, provisioner *provisioning.Provisioner, clk clock.Clock, recorder events.Recorder,
	schedulerOpts []scheduling.Options, candidates ...*Candidate,
	) (scheduling.Results, error) {
	candidateNames := sets.NewString(lo.Map(candidates, func(t *Candidate, i int) string { return t.Name() })...)
	nodes := cluster.DeepCopyNodes()

and makes another deep copy of nodes.

Only place here underlying node is mutated during disruption:

karpenter/pkg/controllers/provisioning/scheduling/existingnode.go

Lines 178 to 179 in d1aacae

	n.HostPortUsage().Add(pod, scheduling.GetHostPorts(pod))
	n.VolumeUsage().Add(pod, volumes)

This means that we likely can get away without copying this every time.

Consistent Snapshot

To address the second problem, we are going to move to taking snapshots of state of nodes instead of deep copy.

This was problematic because we have mutations that occurs during various events against the nodes:

1. Pod bind (updateNodeUsageFromPod, called from UpdatePod) — used to call n.updateForPod(...) directly on the map's live *StateNode.
2. Pod unbind (updateNodeUsageFromPodCompletion, called from DeletePod) — same pattern with n.cleanupForPod(...).
3. Pod moved nodes (cleanupOldBindings) — same cleanupForPod call against the old node when a pod re-binds elsewhere.
4. Nomination (NominateNodeForPod) — used to call n.Nominate(...) directly on the live pointer.
5. Mark/unmark for deletion (MarkForDeletion/UnmarkForDeletion) — used to flip n.markedForDeletion directly (it already cloned via ShallowCopy() for diffing purposes, but the actual
   field write still hit the live node).
6. Partial node/nodeclaim cleanup (cleanupNodeClaim/cleanupNode, called from DeleteNodeClaim/DeleteNode) — this was a bug found during Phase 2, not in the original plan: the "one side
   still exists" branch did c.nodes[id].NodeClaim = nil / c.nodes[id].Node = nil straight on the map's pointer.
7. Reset() — a bulk mutation (reassigns the whole c.nodes map), which needed c.generation++ so Snapshot()'s cache wouldn't serve the pre-Reset data (this was the actual regression bug
   we found and fixed).

So each mutation action needs to create a new version of statenode now (not a deep copy so this is less memory intensive).

Operation	Old (pre-refactor)	Now (copy-on-write)
Pod bind (updateNodeUsageFromPod)	No write-side copy — n.updateForPod(...) mutated the live *StateNode sitting in c.nodes[id] directly	CopyForMutation() — clones podRequests, podLimits, daemonSetRequests, daemonSetLimits, podDisruptionCosts, hostPortUsage, volumeUsage; shares Node/NodeClaim
Pod unbind (updateNodeUsageFromPodCompletion)	No write-side copy — n.cleanupForPod(...) mutated the live node directly	CopyForMutation() — same clone set as above
Pod moved nodes (cleanupOldBindings)	No write-side copy — mutated the old node directly	CopyForMutation() — same clone set as above
Nomination (NominateNodeForPod)	No write-side copy — n.Nominate(...) mutated nominatedUntil directly	ShallowCopy() — copies just the struct; nominatedUntil set on the copy
Mark/Unmark for deletion	ShallowCopy(), but only for diffing — oldNode := n.ShallowCopy() was taken to compute the resource-pool delta, but n.markedForDeletion was still flipped on the live node, not the copy	ShallowCopy(), and it's what gets published — flips markedForDeletion on the copy, swaps it into c.nodes[id]
UpdateNode	Already built fresh — newStateFromNode constructs a brand-new &StateNode{} (unchanged by this refactor)	Same as old — new &StateNode{}; only change is c.generation++ added after the swap
UpdateNodeClaim	Already built fresh — newStateFromNodeClaim constructs a brand-new &StateNode{} (unchanged by this refactor)	Same as old — new &StateNode{}; only change is c.generation++ added
Partial cleanup (cleanupNodeClaim/cleanupNode, other side still exists)	No write-side copy (the actual bug) — oldNode := c.nodes[id].ShallowCopy() was taken for diffing only; c.nodes[id].NodeClaim = nil (or .Node = nil) mutated the live node directly	ShallowCopy(), and it's what gets published — nils the field on the copy, swaps it in
Full removal (cleanupNodeClaim/cleanupNode, nothing left)	delete(c.nodes, id) — no copy needed either way	Same — delete(c.nodes, id), plus c.generation++
Reset()	c.nodes = map[string]*StateNode{} — no copy involved, no generation to invalidate (no cache existed)	Same map reassignment, plus c.generation++ and c.cachedSnap = nil (the bug we found: forgetting this left Snapshot() serving the stale pre-Reset cache)
Every read (DeepCopyNodes() / Snapshot())	DeepCopy() on every node, every call — full clone of Node, NodeClaim, all 5 maps (with per-Quantity clones), both usage trackers	No copy at all on a cache hit — lo.Values(c.nodes), a pointer-slice copy, memoized by generation; cache-miss cost is O(n) pointer copies, not clones

Outcome

Replaces Cluster.DeepCopyNodes()'s full deep-clone-on-every-call with a copy-on-write model: all write paths (pod bind/unbind, mark-for-deletion, nomination, cleanup) now clone-then-swap only the specific
mutable fields they touch, instead of mutating live *StateNodes in place. Reads get a generation-counter-cached Snapshot() that returns a memoized pointer slice when nothing has changed since the last
call, and rebuilds a cheap pointer copy (not a deep clone) otherwise. ExistingNode now clones just its own host-port/volume usage trackers at construction, so SimulateScheduling no longer needs a
caller-provided deep copy at all.

Net effect: snapshot/candidate-construction cost drops from O(n) deep clone to near-O(1) cache hits, at the cost of a modest, bounded increase in per-mutation-event cost on the write path
(bind/unbind/nominate/mark).

New metrics
Added the per-candidate evaluation latency metric. CandidateEvaluationDurationSeconds karpenter_voluntary_disruption_candidate_evaluation_duration_seconds now wraps every SimulateScheduling call site: consolidation.go (single + multi-node), drift.go's per-candidate loop, and validation.go's final revalidation, labeled by reason, consolidation_type, and stage (evaluate vs validate)

How was this change tested?

Perf test:

Benchmark	Baseline	Update	Δ
Snapshot/DeepCopyNodes (5000 nodes)	38.5 ms	3.8 ns	~10,000,000× faster
Snapshot/DeepCopyNodes (400 nodes)	2.69 ms	3.7 ns	~730,000× faster
ClusterDeepCopyNodes (5000, envtest-backed)	17.9 ms	3.7 ns	~4,800,000× faster
CopyForMutation vs DeepCopy (100 pods/node)	69.8 μs (DeepCopy)	18.9 μs (CopyForMutation)	~3.7× cheaper
SimulateScheduling end-to-end (5000 nodes)	83.3 ms	~59–70 ms	~15–29% faster
SimulateScheduling end-to-end (400 nodes)	10.1 ms	8.3–8.9 ms	~12–18% faster
UpdatePod write path, high churn (1000 events/round)	13.8–14.6 ms	22.6–22.8 ms	~1.6× slower (accepted trade-off)
MarkForDeletion / NominateNodeForPod	flat vs. scale	flat vs. scale	no change

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[kubernetes-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: GnatorX
Once this PR has been reviewed and has the lgtm label, please assign jonathan-innis for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kubernetes-prow]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.