Skip to content

Commit 4d25ca2

Browse files
georgehonggeorgehong
committed
Add NFD topology-updater and numa-scheduler modules (#696)
Adds two new OSDC modules for NUMA-aware GPU scheduling: 1. nfd — deploys NFD topology-updater as a DaemonSet on GPU nodes. Publishes NodeResourceTopology CRDs showing per-NUMA-zone resource availability (GPU/CPU/memory per socket). Polls every 15s to stay ahead of the ARC capacity provisioner's 30s interval. 2. numa-scheduler — deploys a secondary kube-scheduler with the NodeResourceTopologyMatch plugin. Reads NRT objects and only places pods on nodes where a single NUMA zone can satisfy the full resource request. Uses MostAllocated scoring to pack small pods together and keep free sockets available for large (4-GPU) jobs. The scheduler-plugins chart is downloaded from the GitHub release (kubernetes-sigs/scheduler-plugins) at deploy time — the project does not publish to a Helm registry. Both modules default to enabled=false in clusters.yaml and are inert until runner definitions set scheduler_name: numa-scheduler on workflow pods. Deploying them has no effect on existing scheduling. Part of #696 ghstack-source-id: b9b30d1 Pull-Request: #716
1 parent 808580f commit 4d25ca2

4 files changed

Lines changed: 223 additions & 0 deletions

File tree

‎osdc/modules/nfd/deploy.sh‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
#
4+
# Deploy NFD topology-updater for NUMA-aware scheduling.
5+
# Called by: just deploy-module <cluster> nfd
6+
#
7+
# Args: $1=cluster-id $2=cluster-name $3=region
8+
#
9+
# Installs the Node Feature Discovery Helm chart with only the
10+
# topology-updater enabled. This publishes NodeResourceTopology CRDs
11+
# that the numa-scheduler reads to make NUMA-aware placement decisions.
12+
13+
CLUSTER="$1"
14+
_CNAME="$2" # unused but required by deploy-module interface
15+
_REGION="$3" # unused but required by deploy-module interface
16+
MODULE_DIR="$(cd "$(dirname "$0")" && pwd)"
17+
REPO_ROOT="${OSDC_ROOT:-$(cd "$MODULE_DIR/../.." && pwd)}"
18+
UPSTREAM_ROOT="${OSDC_UPSTREAM:-$REPO_ROOT}"
19+
20+
# shellcheck source=/dev/null
21+
source "$UPSTREAM_ROOT/scripts/mise-activate.sh"
22+
# shellcheck source=/dev/null
23+
source "$UPSTREAM_ROOT/scripts/helm-upgrade.sh"
24+
25+
CFG="$UPSTREAM_ROOT/scripts/cluster-config.py"
26+
27+
# --- Check if enabled ---
28+
ENABLED=$(uv run "$CFG" "$CLUSTER" nfd.enabled "false")
29+
if [[ "$ENABLED" != "true" ]]; then
30+
echo "NFD disabled for cluster $CLUSTER, skipping."
31+
exit 0
32+
fi
33+
34+
NFD_VERSION=$(uv run "$CFG" "$CLUSTER" nfd.version "0.17.1")
35+
NFD_UPDATE_INTERVAL=$(uv run "$CFG" "$CLUSTER" nfd.update_interval "15s")
36+
37+
echo "Installing NFD topology-updater v${NFD_VERSION}..."
38+
helm_upgrade_if_changed nfd nfd \
39+
--create-namespace \
40+
--history-max 3 \
41+
-f "$MODULE_DIR/helm/values.yaml" \
42+
--set topologyUpdater.updateInterval="${NFD_UPDATE_INTERVAL}" \
43+
--timeout 5m \
44+
--wait \
45+
oci://ghcr.io/kubernetes-sigs/charts/node-feature-discovery \
46+
--version "${NFD_VERSION}"
47+
48+
echo "NFD topology-updater deployed."

‎osdc/modules/nfd/helm/values.yaml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Node Feature Discovery — topology-updater only
2+
# Chart: https://github.com/kubernetes-sigs/node-feature-discovery/tree/master/deployment/helm/node-feature-discovery
3+
#
4+
# We only need the topology-updater component, which publishes
5+
# NodeResourceTopology CRDs per node. The NFD worker (feature detection)
6+
# and master (label reconciliation) are disabled — we don't need node
7+
# feature labels, just per-NUMA-zone resource visibility for the
8+
# numa-scheduler.
9+
10+
master:
11+
enable: false
12+
13+
worker:
14+
enable: false
15+
16+
topologyUpdater:
17+
enable: true
18+
19+
# Poll kubelet PodResources API every 15s (default 60s). Needs to be
20+
# faster than the ARC capacity provisioner interval (30s) so NRT
21+
# objects reflect current GPU allocations before placeholder pods are
22+
# evaluated by the NUMA-aware scheduler.
23+
updateInterval: 15s
24+
25+
# Only run on GPU nodes — non-GPU nodes don't have NUMA topology
26+
# concerns and don't need NRT objects.
27+
nodeSelector:
28+
nvidia.com/gpu: "true"
29+
30+
# Tolerate all GPU node taints so the topology-updater schedules
31+
# at provisioning time, same as other GPU-node DaemonSets.
32+
tolerations:
33+
- key: nvidia.com/gpu
34+
operator: Exists
35+
effect: NoSchedule
36+
- key: node-fleet
37+
operator: Exists
38+
effect: NoSchedule
39+
- key: instance-type
40+
operator: Exists
41+
effect: NoSchedule
42+
- key: git-cache-not-ready
43+
operator: Exists
44+
effect: NoSchedule
45+
46+
resources:
47+
requests:
48+
cpu: 50m
49+
memory: 64Mi
50+
limits:
51+
cpu: 200m
52+
memory: 128Mi
53+
54+
# The topology-updater needs the NRT CRD. The NFD chart installs it
55+
# when topologyUpdater is enabled.
56+
topologyGC:
57+
enable: true

‎osdc/modules/numa-scheduler/deploy.sh‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
#
4+
# Deploy NUMA-aware secondary scheduler (scheduler-plugins).
5+
# Called by: just deploy-module <cluster> numa-scheduler
6+
#
7+
# Args: $1=cluster-id $2=cluster-name $3=region
8+
#
9+
# Downloads the scheduler-plugins Helm chart from the GitHub release
10+
# and installs it as a secondary scheduler named "numa-scheduler" with
11+
# NodeResourceTopologyMatch enabled. Pods that set
12+
# schedulerName: numa-scheduler get NUMA-aware placement.
13+
14+
CLUSTER="$1"
15+
_CNAME="$2" # unused but required by deploy-module interface
16+
_REGION="$3" # unused but required by deploy-module interface
17+
MODULE_DIR="$(cd "$(dirname "$0")" && pwd)"
18+
REPO_ROOT="${OSDC_ROOT:-$(cd "$MODULE_DIR/../.." && pwd)}"
19+
UPSTREAM_ROOT="${OSDC_UPSTREAM:-$REPO_ROOT}"
20+
21+
# shellcheck source=/dev/null
22+
source "$UPSTREAM_ROOT/scripts/mise-activate.sh"
23+
# shellcheck source=/dev/null
24+
source "$UPSTREAM_ROOT/scripts/helm-upgrade.sh"
25+
26+
CFG="$UPSTREAM_ROOT/scripts/cluster-config.py"
27+
28+
# --- Check if enabled ---
29+
ENABLED=$(uv run "$CFG" "$CLUSTER" numa_scheduler.enabled "false")
30+
if [[ "$ENABLED" != "true" ]]; then
31+
echo "numa-scheduler disabled for cluster $CLUSTER, skipping."
32+
exit 0
33+
fi
34+
35+
CHART_VERSION=$(uv run "$CFG" "$CLUSTER" numa_scheduler.chart_version "0.34.7")
36+
SCHEDULER_REPLICAS=$(uv run "$CFG" "$CLUSTER" numa_scheduler.replicas "2")
37+
38+
# --- Download chart from GitHub release ---
39+
CHART_TGZ="scheduler-plugins-${CHART_VERSION}.tgz"
40+
CHART_DIR=$(mktemp -d)
41+
trap 'rm -rf "$CHART_DIR"' EXIT
42+
43+
if [[ ! -f "${CHART_DIR}/${CHART_TGZ}" ]]; then
44+
echo "Downloading scheduler-plugins chart v${CHART_VERSION}..."
45+
gh release download "v${CHART_VERSION}" \
46+
--repo kubernetes-sigs/scheduler-plugins \
47+
--pattern "${CHART_TGZ}" \
48+
--dir "$CHART_DIR"
49+
fi
50+
51+
echo "Installing numa-scheduler (scheduler-plugins v${CHART_VERSION})..."
52+
helm_upgrade_if_changed numa-scheduler numa-scheduler \
53+
--create-namespace \
54+
--history-max 3 \
55+
-f "$MODULE_DIR/helm/values.yaml" \
56+
--set scheduler.replicaCount="${SCHEDULER_REPLICAS}" \
57+
--timeout 5m \
58+
--wait \
59+
"${CHART_DIR}/${CHART_TGZ}"
60+
61+
echo "numa-scheduler deployed."

‎osdc/modules/numa-scheduler/helm/values.yaml‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Scheduler-plugins — NUMA-aware secondary scheduler
2+
# Chart source: https://github.com/kubernetes-sigs/scheduler-plugins
3+
# (vendored from manifests/install/charts/as-a-second-scheduler/)
4+
#
5+
# Deploys a second kube-scheduler named "numa-scheduler" with the
6+
# NodeResourceTopologyMatch plugin enabled. Pods that set
7+
# schedulerName: numa-scheduler will be filtered/scored by per-NUMA-zone
8+
# resource availability (read from NodeResourceTopology CRDs published
9+
# by the NFD topology-updater).
10+
#
11+
# Nothing uses this scheduler until runner definitions set
12+
# scheduler_name: numa-scheduler — deploying it is a no-op.
13+
14+
scheduler:
15+
name: numa-scheduler
16+
image: registry.k8s.io/scheduler-plugins/kube-scheduler:v0.34.7
17+
replicaCount: 2
18+
leaderElect: true
19+
20+
# Run on base-infrastructure nodes alongside other control plane components.
21+
nodeSelector:
22+
role: base-infrastructure
23+
24+
tolerations:
25+
- key: CriticalAddonsOnly
26+
operator: Equal
27+
value: "true"
28+
effect: NoSchedule
29+
30+
resources:
31+
requests:
32+
cpu: 200m
33+
memory: 256Mi
34+
limits:
35+
cpu: 500m
36+
memory: 512Mi
37+
38+
# The controller is needed for Coscheduling/CapacityScheduling plugins.
39+
# We only use NodeResourceTopologyMatch, so disable it.
40+
controller:
41+
replicaCount: 0
42+
43+
# Only enable NodeResourceTopologyMatch — we don't need Coscheduling,
44+
# CapacityScheduling, or other plugins.
45+
plugins:
46+
enabled:
47+
- NodeResourceTopologyMatch
48+
disabled: []
49+
50+
pluginConfig:
51+
- name: NodeResourceTopologyMatch
52+
args:
53+
scoringStrategy:
54+
# Pack pods onto already-busy NUMA zones so other zones stay
55+
# fully free for large (4-GPU) requests. Without this, small
56+
# pods spread across sockets and fragment both.
57+
type: MostAllocated

0 commit comments

Comments
 (0)