Skip to content

Commit b8cfbd2

Browse files
committed
Add NFD topology-updater and numa-scheduler modules (#696)
Adds two new OSDC modules for NUMA-aware GPU scheduling: 1. nfd — deploys NFD topology-updater as a DaemonSet on GPU nodes. Publishes NodeResourceTopology CRDs showing per-NUMA-zone resource availability (GPU/CPU/memory per socket). Polls every 15s to stay ahead of the ARC capacity provisioner's 30s interval. 2. numa-scheduler — deploys a secondary kube-scheduler with the NodeResourceTopologyMatch plugin. Reads NRT objects and only places pods on nodes where a single NUMA zone can satisfy the full resource request. Uses MostAllocated scoring to pack small pods together and keep free sockets available for large (4-GPU) jobs. The scheduler-plugins chart is downloaded from the GitHub release (kubernetes-sigs/scheduler-plugins) at deploy time — the project does not publish to a Helm registry. Both modules default to enabled=false in clusters.yaml and are inert until runner definitions set scheduler_name: numa-scheduler on workflow pods. Deploying them has no effect on existing scheduling. Part of #696 ghstack-source-id: c8ce9d0 Pull-Request: #716
1 parent e056553 commit b8cfbd2

8 files changed

Lines changed: 420 additions & 0 deletions

File tree

osdc/modules/nfd/deploy.sh

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
#
4+
# Deploy NFD topology-updater for NUMA-aware scheduling.
5+
# Called by: just deploy-module <cluster> nfd
6+
#
7+
# Args: $1=cluster-id $2=cluster-name $3=region
8+
#
9+
# Installs the Node Feature Discovery Helm chart with only the
10+
# topology-updater enabled. This publishes NodeResourceTopology CRDs
11+
# that the numa-scheduler reads to make NUMA-aware placement decisions.
12+
13+
CLUSTER="$1"
14+
_CNAME="$2" # unused but required by deploy-module interface
15+
_REGION="$3" # unused but required by deploy-module interface
16+
MODULE_DIR="$(cd "$(dirname "$0")" && pwd)"
17+
REPO_ROOT="${OSDC_ROOT:-$(cd "$MODULE_DIR/../.." && pwd)}"
18+
UPSTREAM_ROOT="${OSDC_UPSTREAM:-$REPO_ROOT}"
19+
20+
# shellcheck source=/dev/null
21+
source "$UPSTREAM_ROOT/scripts/mise-activate.sh"
22+
# shellcheck source=/dev/null
23+
source "$UPSTREAM_ROOT/scripts/helm-upgrade.sh"
24+
25+
CFG="$UPSTREAM_ROOT/scripts/cluster-config.py"
26+
27+
# --- Check if enabled ---
28+
ENABLED=$(uv run "$CFG" "$CLUSTER" nfd.enabled "false")
29+
if [[ "$ENABLED" != "true" ]]; then
30+
echo "NFD disabled for cluster $CLUSTER, skipping."
31+
exit 0
32+
fi
33+
34+
NFD_VERSION=$(uv run "$CFG" "$CLUSTER" nfd.version "0.17.1")
35+
NFD_UPDATE_INTERVAL=$(uv run "$CFG" "$CLUSTER" nfd.update_interval "15s")
36+
37+
helm repo add nfd https://kubernetes-sigs.github.io/node-feature-discovery/charts 2>/dev/null || true
38+
helm repo update nfd
39+
40+
echo "Installing NFD topology-updater v${NFD_VERSION}..."
41+
helm_upgrade_if_changed nfd nfd \
42+
--create-namespace \
43+
--history-max 3 \
44+
-f "$MODULE_DIR/helm/values.yaml" \
45+
--set topologyUpdater.updateInterval="${NFD_UPDATE_INTERVAL}" \
46+
--timeout 5m \
47+
--wait \
48+
nfd/node-feature-discovery \
49+
--version "${NFD_VERSION}"
50+
51+
echo "NFD topology-updater deployed."

osdc/modules/nfd/helm/values.yaml

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Node Feature Discovery — topology-updater only
2+
# Chart: https://github.com/kubernetes-sigs/node-feature-discovery/tree/master/deployment/helm/node-feature-discovery
3+
#
4+
# We only need the topology-updater component, which publishes
5+
# NodeResourceTopology CRDs per node. The NFD worker (feature detection)
6+
# and master (label reconciliation) are disabled — we don't need node
7+
# feature labels, just per-NUMA-zone resource visibility for the
8+
# numa-scheduler.
9+
10+
master:
11+
enable: false
12+
13+
worker:
14+
enable: false
15+
16+
topologyUpdater:
17+
enable: true
18+
19+
# Poll kubelet PodResources API every 15s (default 60s). Needs to be
20+
# faster than the ARC capacity provisioner interval (30s) so NRT
21+
# objects reflect current GPU allocations before placeholder pods are
22+
# evaluated by the NUMA-aware scheduler.
23+
updateInterval: 15s
24+
25+
# Only run on GPU nodes — non-GPU nodes don't have NUMA topology
26+
# concerns and don't need NRT objects.
27+
nodeSelector:
28+
nvidia.com/gpu: "true"
29+
30+
# Tolerate all GPU node taints so the topology-updater schedules
31+
# at provisioning time, same as other GPU-node DaemonSets.
32+
tolerations:
33+
- key: nvidia.com/gpu
34+
operator: Exists
35+
effect: NoSchedule
36+
- key: node-fleet
37+
operator: Exists
38+
effect: NoSchedule
39+
- key: instance-type
40+
operator: Exists
41+
effect: NoSchedule
42+
- key: git-cache-not-ready
43+
operator: Exists
44+
effect: NoSchedule
45+
46+
resources:
47+
requests:
48+
cpu: 50m
49+
memory: 64Mi
50+
limits:
51+
cpu: 200m
52+
memory: 128Mi
53+
54+
# The topology-updater needs the NRT CRD. The NFD chart installs it
55+
# when topologyUpdater is enabled.
56+
topologyGC:
57+
enable: true
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
from smoke_conftest import * # noqa: F403
Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
"""Smoke tests for NFD (Node Feature Discovery) topology-updater.
2+
3+
Validates that the NFD namespace exists, the Helm release is deployed,
4+
the topology-updater DaemonSet is healthy on GPU nodes, and
5+
NodeResourceTopology CRDs are being published.
6+
"""
7+
8+
from __future__ import annotations
9+
10+
import subprocess
11+
12+
import pytest
13+
from helpers import (
14+
assert_daemonset_healthy,
15+
find_helm_release,
16+
run_kubectl,
17+
)
18+
19+
pytestmark = [pytest.mark.live]
20+
21+
NAMESPACE = "nfd"
22+
23+
24+
# ============================================================================
25+
# Namespace
26+
# ============================================================================
27+
28+
29+
class TestNFDNamespace:
30+
"""Verify NFD namespace exists."""
31+
32+
def test_namespace_exists(self, all_namespaces: dict) -> None:
33+
ns_names = [ns["metadata"]["name"] for ns in all_namespaces.get("items", [])]
34+
assert NAMESPACE in ns_names, f"Namespace '{NAMESPACE}' not found"
35+
36+
37+
# ============================================================================
38+
# Helm Release
39+
# ============================================================================
40+
41+
42+
class TestNFDHelm:
43+
"""Verify NFD Helm release is deployed."""
44+
45+
def test_helm_release_deployed(self, all_helm_releases: list[dict]) -> None:
46+
release = find_helm_release(all_helm_releases, "nfd", namespace=NAMESPACE)
47+
assert release is not None, "Helm release 'nfd' not found in 'nfd' namespace"
48+
status = release.get("status", "")
49+
assert status == "deployed", f"NFD Helm release status is '{status}', expected 'deployed'"
50+
51+
52+
# ============================================================================
53+
# Topology-updater DaemonSet
54+
# ============================================================================
55+
56+
57+
class TestTopologyUpdater:
58+
"""Verify topology-updater DaemonSet is healthy on GPU nodes.
59+
60+
allow_zero=True because the cluster may not have any GPU nodes
61+
provisioned at test time (e.g. staging with no active GPU jobs).
62+
"""
63+
64+
def test_topology_updater_healthy(self, all_daemonsets: dict, all_nodes: dict) -> None:
65+
assert_daemonset_healthy(
66+
all_daemonsets,
67+
all_nodes,
68+
NAMESPACE,
69+
name_contains="topology-updater",
70+
allow_zero=True,
71+
)
72+
73+
74+
# ============================================================================
75+
# NodeResourceTopology CRD
76+
# ============================================================================
77+
78+
79+
class TestNodeResourceTopology:
80+
"""Verify NodeResourceTopology CRD is installed.
81+
82+
The NRT CRD is the key output of NFD topology-updater — without it
83+
the numa-scheduler has no NUMA visibility. The CRD is installed by
84+
the NFD Helm chart when topologyUpdater is enabled.
85+
"""
86+
87+
def test_nrt_crd_exists(self) -> None:
88+
try:
89+
result = run_kubectl(
90+
["get", "crd", "noderesourcetopologies.topology.node.k8s.io", "-o", "json"],
91+
)
92+
except subprocess.CalledProcessError:
93+
pytest.fail("NodeResourceTopology CRD not found — NFD chart may not have installed it")
94+
name = result.get("metadata", {}).get("name", "")
95+
assert "noderesourcetopologies" in name
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
#
4+
# Deploy NUMA-aware secondary scheduler (scheduler-plugins).
5+
# Called by: just deploy-module <cluster> numa-scheduler
6+
#
7+
# Args: $1=cluster-id $2=cluster-name $3=region
8+
#
9+
# Downloads the scheduler-plugins Helm chart from the GitHub release
10+
# and installs it as a secondary scheduler named "numa-scheduler" with
11+
# NodeResourceTopologyMatch enabled. Pods that set
12+
# schedulerName: numa-scheduler get NUMA-aware placement.
13+
14+
CLUSTER="$1"
15+
_CNAME="$2" # unused but required by deploy-module interface
16+
_REGION="$3" # unused but required by deploy-module interface
17+
MODULE_DIR="$(cd "$(dirname "$0")" && pwd)"
18+
REPO_ROOT="${OSDC_ROOT:-$(cd "$MODULE_DIR/../.." && pwd)}"
19+
UPSTREAM_ROOT="${OSDC_UPSTREAM:-$REPO_ROOT}"
20+
21+
# shellcheck source=/dev/null
22+
source "$UPSTREAM_ROOT/scripts/mise-activate.sh"
23+
# shellcheck source=/dev/null
24+
source "$UPSTREAM_ROOT/scripts/helm-upgrade.sh"
25+
26+
CFG="$UPSTREAM_ROOT/scripts/cluster-config.py"
27+
28+
# --- Check if enabled ---
29+
ENABLED=$(uv run "$CFG" "$CLUSTER" numa_scheduler.enabled "false")
30+
if [[ "$ENABLED" != "true" ]]; then
31+
echo "numa-scheduler disabled for cluster $CLUSTER, skipping."
32+
exit 0
33+
fi
34+
35+
CHART_VERSION=$(uv run "$CFG" "$CLUSTER" numa_scheduler.chart_version "0.34.7")
36+
SCHEDULER_REPLICAS=$(uv run "$CFG" "$CLUSTER" numa_scheduler.replicas "2")
37+
38+
# --- Download chart from GitHub release ---
39+
CHART_TGZ="scheduler-plugins-${CHART_VERSION}.tgz"
40+
CHART_DIR=$(mktemp -d)
41+
trap 'rm -rf "$CHART_DIR"' EXIT
42+
43+
CHART_URL="https://github.com/kubernetes-sigs/scheduler-plugins/releases/download/v${CHART_VERSION}/${CHART_TGZ}"
44+
echo "Downloading scheduler-plugins chart v${CHART_VERSION}..."
45+
curl -fsSL "$CHART_URL" -o "${CHART_DIR}/${CHART_TGZ}"
46+
47+
echo "Installing numa-scheduler (scheduler-plugins v${CHART_VERSION})..."
48+
helm_upgrade_if_changed numa-scheduler numa-scheduler \
49+
--create-namespace \
50+
--history-max 3 \
51+
-f "$MODULE_DIR/helm/values.yaml" \
52+
--set scheduler.replicaCount="${SCHEDULER_REPLICAS}" \
53+
--timeout 5m \
54+
--wait \
55+
"${CHART_DIR}/${CHART_TGZ}"
56+
57+
echo "numa-scheduler deployed."
Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Scheduler-plugins — NUMA-aware secondary scheduler
2+
# Chart source: https://github.com/kubernetes-sigs/scheduler-plugins
3+
# (vendored from manifests/install/charts/as-a-second-scheduler/)
4+
#
5+
# Deploys a second kube-scheduler named "numa-scheduler" with the
6+
# NodeResourceTopologyMatch plugin enabled. Pods that set
7+
# schedulerName: numa-scheduler will be filtered/scored by per-NUMA-zone
8+
# resource availability (read from NodeResourceTopology CRDs published
9+
# by the NFD topology-updater).
10+
#
11+
# Nothing uses this scheduler until runner definitions set
12+
# scheduler_name: numa-scheduler — deploying it is a no-op.
13+
14+
scheduler:
15+
name: numa-scheduler
16+
image: registry.k8s.io/scheduler-plugins/kube-scheduler:v0.34.7
17+
replicaCount: 2
18+
leaderElect: true
19+
20+
# Run on base-infrastructure nodes alongside other control plane components.
21+
nodeSelector:
22+
role: base-infrastructure
23+
24+
tolerations:
25+
- key: CriticalAddonsOnly
26+
operator: Equal
27+
value: "true"
28+
effect: NoSchedule
29+
30+
resources:
31+
requests:
32+
cpu: 200m
33+
memory: 256Mi
34+
limits:
35+
cpu: 500m
36+
memory: 512Mi
37+
38+
# The controller is needed for Coscheduling/CapacityScheduling plugins.
39+
# We only use NodeResourceTopologyMatch, so disable it.
40+
controller:
41+
replicaCount: 0
42+
43+
# Only enable NodeResourceTopologyMatch — we don't need Coscheduling,
44+
# CapacityScheduling, or other plugins.
45+
plugins:
46+
enabled:
47+
- NodeResourceTopologyMatch
48+
disabled: []
49+
50+
pluginConfig:
51+
- name: NodeResourceTopologyMatch
52+
args:
53+
scoringStrategy:
54+
# Pack pods onto already-busy NUMA zones so other zones stay
55+
# fully free for large (4-GPU) requests. Without this, small
56+
# pods spread across sockets and fragment both.
57+
type: MostAllocated
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
from smoke_conftest import * # noqa: F403

0 commit comments

Comments
 (0)