feat(scheduler): Implement node scoring in NUMA plugin (#1965)

Signed-off-by: itsomri <omric@nvidia.com>

Author: itsomri

.changes/unreleased/added-20260723-175006.yaml

@@ -0,0 +1,3 @@
+kind: Added
+body: |-
+  NUMA-aware node scoring preferring fewest-NUMA-zone placement

docs/developer/designs/numa-topology/README.md

@@ -601,38 +601,59 @@ regardless; Appendix A is the in-plugin fallback if the assumption proves insuff
 
 ## v2: Optimization & scoring
 
-v1 decides *feasibility* — can this node host the pod without a `TopologyAffinityError`. v2
-decides *which feasible node is best*, via a node score (`AddNodeOrderFn`, a new band in
-`scores/scores.go`). It reuses v1's evaluators and per-zone model unchanged: it only **ranks**
-nodes, never alters the admit decision.
+v2 adds scoring based on NUMA placement: nodes that can fit a NUMA-sensitive task in fewer zones
+are ranked higher. Nodes that can't admit the task are ranked lower. This enables us to support
+`best-effort` mode better.
+
+The upstream scheduler numa plugin supports different scoring strategies - `LeastNUMANodes`, 
+`BalancedAllocation`, `LeastAllocated` and `MostAllocated` - the last three are only relevant to 
+`single-numa-node` mode. `LeastNUMANodes` seems the most relevant for our use cases, but we can
+support more modes, and welcome community feedback on this.
 
 ### What scoring adds
 
 - **Optimize `best-effort` performance.** On a `best-effort` node the kubelet never rejects — it
   silently runs the pod *unaligned* when it can't fit a NUMA node, costing throughput. v1 does
   nothing for `best-effort` (there is no admission error to prevent). v2 **scores** `best-effort`
-  nodes by whether the pod's resources *can* be aligned there, steering it toward a node where
-  the kubelet's best-effort alignment will actually succeed — turning a silent performance loss
-  into a good placement. This is the primary motivation for v2.
-- **Prefer tighter, less-fragmented fit** on feasible `single-numa-node` / `restricted` nodes, so
-  later pods still find aligned room, and multi-NUMA pods span the fewest zones.
-
-### Scoring strategies
-
-Reusing the upstream NodeResourceTopology scoring vocabulary, computed over the plugin's per-zone
-model:
-
-- **LeastNUMANodes** (policy-agnostic) — prefer nodes where the pod spans the fewest NUMA nodes
-  (ideally one). This is the core `best-effort` steering and the multi-NUMA-span minimizer.
-- **LeastAllocated / MostAllocated / BalancedAllocation** — spread vs. bin-pack vs. balance
-  per-zone utilization, for fragmentation control on the aligned policies; selectable via config.
+  nodes by how few zones the pod's resources *can* be aligned to there, steering it toward a node
+  where the kubelet's best-effort alignment will actually succeed. This is the primary motivation for v2.
+- **Prefer tighter, fewer-zone fit** on feasible `restricted` nodes, where the kubelet forces the
+  pod to span its preferred width `w` (which differs per node by per-zone `Allocatable`): rank
+  nodes by `w` so a pod that aligns to one zone on node A beats spanning two on node B.
+- **Sink infeasible nodes so the predicate short-circuits.** `OrderedNodesByTask` scores *every*
+  candidate node — including ones the predicate will reject — and the action then runs `FittingNode`
+  (the predicate) lazily **in score order**, stopping at the first fit. So ranking a node the
+  kubelet can't NUMA-align *below* one it can makes the predicate hit a feasible node first and skip
+  evaluating the infeasible ones. This applies to **`single-numa-node`** too: its feasible span is
+  always 1, but feasibility itself is a ranking signal, so scoring is *not* a no-op there — it
+  front-loads the alignable nodes. (Correctness still rests on the predicate; the score only reorders.)
+
+### The fewest-zones span, per policy
+
+Scoring routes every candidate node through one reusable function, `alignmentSpan(task, node) →
+(zones int, aligned bool)`, and the score is a function of **both** outputs: `aligned=false` sinks
+the node (worst score), and among aligned nodes fewer `zones` scores higher.
+
+| Policy | `alignmentSpan` when aligned | `aligned=false` when | Predicate outcome |
+| --- | --- | --- | --- |
+| `single-numa-node` | `1` | no single zone fits by `Available` | rejects (filtered) |
+| `restricted` | the forced preferred width `w` | preferred widths disagree, or no width-`w` mask fits | rejects (filtered) |
+| `best-effort` | greedy narrowest zone mask that fits by `Available` (width = span) | even all N zones can't cover the request (pod runs unaligned) | passes (best-effort never rejects) |
+
+For the two rejecting policies, `aligned` is the *same bit the predicate computes*, so a sunk node
+is one the predicate would filter — the score just reorders the funnel. For `best-effort`,
+`aligned=false` is the unaligned case: still selectable (worst-but-finite score), because
+`best-effort` offers no other node any guarantee.
 
 ### Notes
 
 - Scoring runs on the same predicted per-zone state as v1, so the prediction caveats carry over —
   but a score is only a *preference*, so a misprediction costs ranking quality, never correctness.
-- `best-effort` scoring is the one place the plugin touches `best-effort` nodes at all; v1 leaves
-  them untouched, and the admit decision for `single-numa-node` / `restricted` is unchanged.
+- The span metric is pure zone-count and policy-agnostic, so span-1 scores identically across
+  `single-numa-node`, `restricted`, and `best-effort` — a mixed-policy candidate set ranks coherently.
+- **No NUMA-distance awareness in v2.** The score is zone *count* only; inter-zone distance
+  (`Zone.Costs`) is not ingested and no scoring seam is reserved for it. A distance metric (e.g. a
+  v3 "minimax") would be a separate axis added later if pursued.
 
 ## v3: Pod-level NUMA policy (scheduler-enforced)
 

pkg/scheduler/plugins/numa/evaluator.go

@@ -4,6 +4,7 @@
 package numa
 
 import (
+	"math"
 	"sort"
 
 	v1 "k8s.io/api/core/v1"
@@ -14,8 +15,11 @@ import (
 	"github.com/kai-scheduler/KAI-scheduler/pkg/scheduler/api/resource_info"
 )
 
-// stackZones bounds the mask/scratch stack buffers; nodes with more NUMA zones fall back to heap.
-const stackZones = 16
+// stackZones and stackAware bound the mask/scratch stack buffers; larger nodes fall back to heap.
+const (
+	stackZones = 16
+	stackAware = 16
+)
 
 // zoneAllocation accumulates, per zone index, the amounts to place there (as a ResourceVector delta).
 // placementFromAllocation materializes it into a pod_info.NUMAPlacement.
@@ -49,29 +53,31 @@ func (pp *numaPlugin) alignedAware(task *pod_info.PodInfo, node *node_info.NodeI
 	return out
 }
 
-// allocatable reports whether the kubelet Topology Manager would align the task on the node. A task
-// the plugin does not constrain passes through as true.
+// allocatable reports whether the kubelet Topology Manager would align the task on the node (the
+// predicate). A task the plugin does not filter passes through as true.
 func (pp *numaPlugin) allocatable(task *pod_info.PodInfo, node *node_info.NodeInfo) bool {
+	if node == nil || !pp.shouldFilter(task, node.NumaTopology) {
+		return true
+	}
 	return pp.solveTask(task, node, nil)
 }
 
 // evaluate returns the task's expected per-zone allocation on the node (nil for a task the plugin
-// does not constrain). Used by the placement path; the predicate uses allocatable (feasibility only).
+// does not account for). Used by the placement and scoring paths.
 func (pp *numaPlugin) evaluate(task *pod_info.PodInfo, node *node_info.NodeInfo) (zoneAllocation, bool) {
+	if node == nil || !pp.shouldScore(task, node.NumaTopology) {
+		return nil, true
+	}
 	alloc := zoneAllocation{}
 	if !pp.solveTask(task, node, alloc) {
 		return nil, false
 	}
 	return alloc, true
 }
 
-// solveTask resolves the task's requests and scope for the node and runs solve. A task the plugin
-// does not constrain passes through as admitted. When alloc is non-nil, solve records the placement
-// into it; when nil, it only decides feasibility (zero-allocation).
+// solveTask runs the evaluator on a node the caller has already gated. When alloc is non-nil, solve
+// records the placement into it; when nil, it only decides feasibility (zero-allocation).
 func (pp *numaPlugin) solveTask(task *pod_info.PodInfo, node *node_info.NodeInfo, alloc zoneAllocation) bool {
-	if node == nil || !pp.shouldHandle(task, node.NumaTopology) {
-		return true
-	}
 	topo := node.NumaTopology
 	aware := pp.alignedAware(task, node)
 	concurrent, serial := pp.numaRequestsFor(task, topo.VectorMap).forScope(topo.Scope)
@@ -118,10 +124,14 @@ func solve(topo *node_info.NumaTopology, aware []int, concurrent, serial []resou
 // feasibleMask picks the mask the policy's evaluator would choose for one request, under the current
 // availability view (Available minus consumed).
 func feasibleMask(topo *node_info.NumaTopology, aware []int, req resource_info.ResourceVector, consumed []float64, width int, maskBuf []int) ([]int, bool) {
-	if topo.Policy == node_info.TopologyPolicySingleNUMANode {
+	switch topo.Policy {
+	case node_info.TopologyPolicySingleNUMANode:
 		return singleNUMAEvaluator{}.fit(topo, aware, req, consumed, width, maskBuf)
+	case node_info.TopologyPolicyBestEffort:
+		return bestEffortEvaluator{}.fit(topo, aware, req, consumed, width, maskBuf)
+	default:
+		return restrictedEvaluator{}.fit(topo, aware, req, consumed, width, maskBuf)
 	}
-	return restrictedEvaluator{}.fit(topo, aware, req, consumed, width, maskBuf)
 }
 
 // singleNUMAEvaluator (single-numa-node) requires each request to fit entirely within one NUMA zone,
@@ -137,6 +147,79 @@ func (singleNUMAEvaluator) fit(topo *node_info.NumaTopology, aware []int, req re
 	return nil, false
 }
 
+// bestEffortEvaluator implements the best-effort policy, which never rejects, so its result feeds
+// scoring and the in-cycle ledger, never an admit decision.
+type bestEffortEvaluator struct{}
+
+// fit greedily grows a zone mask until it covers the request, returning the mask and whether it fits:
+//  1. Start with no zones; the unmet demand is the full request.
+//  2. Each round, add the unused zone that covers the most unmet demand (the sum, over
+//     still-needed resources, of min(remaining, available in that zone)), then subtract that zone's
+//     availability from the demand.
+//  3. Stop when every resource is met (fits) or no remaining zone can help (does not fit).
+//
+// The number of zones picked is the span. Taking the most-covering zone first yields the fewest
+// zones exactly when one resource dominates.
+func (bestEffortEvaluator) fit(topo *node_info.NumaTopology, aware []int, req resource_info.ResourceVector, consumed []float64, width int, maskBuf []int) ([]int, bool) {
+	var remArr [stackAware]float64
+	remaining := remArr[:0]
+	if len(aware) > stackAware {
+		remaining = make([]float64, 0, len(aware))
+	}
+	for _, idx := range aware {
+		remaining = append(remaining, req.Get(idx))
+	}
+	var usedArr [stackZones]bool
+	used := usedArr[:]
+	if len(topo.Zones) > stackZones {
+		used = make([]bool, len(topo.Zones))
+	}
+
+	mask := maskBuf[:0]
+	for len(mask) < len(topo.Zones) {
+		if allMet(remaining) {
+			return mask, true
+		}
+		best, bestCover := -1, 0.0
+		for z := range topo.Zones {
+			if used[z] {
+				continue
+			}
+			cover := 0.0
+			for i, idx := range aware {
+				if remaining[i] <= 0 {
+					continue
+				}
+				cover += math.Min(remaining[i], availableAt(topo, consumed, width, z, idx))
+			}
+			if cover > bestCover {
+				best, bestCover = z, cover
+			}
+		}
+		if best < 0 {
+			break
+		}
+		used[best] = true
+		mask = append(mask, best)
+		for i, idx := range aware {
+			remaining[i] -= availableAt(topo, consumed, width, best, idx)
+		}
+	}
+	if allMet(remaining) {
+		return mask, true
+	}
+	return nil, false
+}
+
+func allMet(remaining []float64) bool {
+	for _, r := range remaining {
+		if r > 0 {
+			return false
+		}
+	}
+	return true
+}
+
 // restrictedEvaluator reproduces the kubelet hint merge: all per-resource preferred widths (from
 // static Allocatable) must agree, and a mask of that width must satisfy every resource against
 // Available. single-numa-node is the width==1 case.