Testing Multi-Node NVLink support on GB200

[klueska]

Prerequisites:

• Multi-Node GB200 cluster with at least 2 GPU nodes and Multi-Node NVLink connections between all GPUs on all nodes
• K8s 1.32 (or later) is up and running with DRA and CDI enabled on all nodes

Setup GPU Operator and install DRA driver

Add the NVIDIA helm repository:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \
    && helm repo update

Install the the GPU Operator and the NVIDIA DRA Driver for GPUs:

With HOST managed drivers

Prerequisites:

• R570 (or later) driver is installed on all GPU nodes
• IMEX packages are installed on GPU nodes with systemd service either non-existent, or disabled and masked

systemctl disable --now nvidia-imex.service && systemctl mask nvidia-imex.service

Install the GPU Operator:

helm install gpu-operator nvidia/gpu-operator \
    --version="v25.3.2" \
    --create-namespace \
    --namespace gpu-operator \
    --set cdi.enabled=true \
    --set driver.enabled=false

Install the DRA Driver for GPUs:

helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
    --version="25.3.0" \
    --create-namespace \
    --namespace nvidia-dra-driver-gpu \
    --set nvidiaDriverRoot=/ \
    --set resources.gpus.enabled=false

With OPERATOR managed drivers

Install the GPU Operator:

helm install gpu-operator nvidia/gpu-operator \
    --version="v25.3.2" \
    --create-namespace \
    --namespace gpu-operator \
    --set cdi.enabled=true

Install the DRA Driver for GPUs:

helm install nvidia-dra-driver-gpu nvidia/nvidia-dra-driver-gpu \
    --version="25.3.0" \
    --create-namespace \
    --namespace nvidia-dra-driver-gpu \
    --set nvidiaDriverRoot=/run/nvidia/driver \
    --set resources.gpus.enabled=false

Validate GPU Operator and DRA driver running

Validate that all GPU Operator components are running and in a Ready state:

$ kubectl get pod -n gpu-operator
NAME                                                         READY   STATUS      RESTARTS   AGE
gpu-feature-discovery-824ts                                  1/1     Running     0          111s
gpu-feature-discovery-b5r5m                                  1/1     Running     0          111s
gpu-feature-discovery-rzzjm                                  1/1     Running     0          111s
gpu-feature-discovery-wx2qd                                  1/1     Running     0          111s
gpu-operator-759dd5976b-m2nw4                                1/1     Running     0          16m
gpu-operator-node-feature-discovery-gc-fc549bd94-czgsv       1/1     Running     0          16m
gpu-operator-node-feature-discovery-master-b4bb855b7-rjmrm   1/1     Running     0          16m
gpu-operator-node-feature-discovery-worker-2wh9x             1/1     Running     0          16m
gpu-operator-node-feature-discovery-worker-bhvpc             1/1     Running     0          16m
gpu-operator-node-feature-discovery-worker-llhrn             1/1     Running     0          16m
gpu-operator-node-feature-discovery-worker-p68cf             1/1     Running     0          16m
nvidia-container-toolkit-daemonset-9fm8m                     1/1     Running     0          15m
nvidia-container-toolkit-daemonset-ckp7k                     1/1     Running     0          15m
nvidia-container-toolkit-daemonset-r6xb7                     1/1     Running     0          15m
nvidia-container-toolkit-daemonset-tmvcb                     1/1     Running     0          15m
nvidia-cuda-validator-5glv9                                  0/1     Completed   0          15m
nvidia-cuda-validator-94vzp                                  0/1     Completed   0          15m
nvidia-cuda-validator-h88m6                                  0/1     Completed   0          15m
nvidia-cuda-validator-xsb6g                                  0/1     Completed   0          15m
nvidia-dcgm-exporter-l5g5g                                   1/1     Running     0          15m
nvidia-dcgm-exporter-s2x4s                                   1/1     Running     0          15m
nvidia-dcgm-exporter-tlwjw                                   1/1     Running     0          15m
nvidia-dcgm-exporter-zwn9q                                   1/1     Running     0          15m
nvidia-device-plugin-daemonset-hwkt5                         1/1     Running     0          15m
nvidia-device-plugin-daemonset-nkrj8                         1/1     Running     0          15m
nvidia-device-plugin-daemonset-pxn2b                         1/1     Running     0          15m
nvidia-device-plugin-daemonset-rpz27                         1/1     Running     0          15m
nvidia-mig-manager-9bpnq                                     1/1     Running     0          15m
nvidia-mig-manager-9vzrd                                     1/1     Running     0          15m
nvidia-mig-manager-mhdjn                                     1/1     Running     0          15m
nvidia-mig-manager-v65xc                                     1/1     Running     0          15m
nvidia-operator-validator-2gmf6                              1/1     Running     0          15m
nvidia-operator-validator-6ts67                              1/1     Running     0          15m
nvidia-operator-validator-qnwml                              1/1     Running     0          15m
nvidia-operator-validator-rk6xk                              1/1     Running     0          15m

Validate that the DRA driver components are running and in a Ready state:

$ kubectl get pod -n nvidia-dra-driver-gpu
NAME                                                           READY   STATUS    RESTARTS   AGE
nvidia-dra-driver-k8s-dra-driver-controller-67cb99d84b-5q7kj   1/1     Running   0          7m26s
nvidia-dra-driver-k8s-dra-driver-kubelet-plugin-7kdg9          1/1     Running   0          7m27s
nvidia-dra-driver-k8s-dra-driver-kubelet-plugin-bd6gn          1/1     Running   0          7m27s
nvidia-dra-driver-k8s-dra-driver-kubelet-plugin-bzm6p          1/1     Running   0          7m26s
nvidia-dra-driver-k8s-dra-driver-kubelet-plugin-xjm4p          1/1     Running   0          7m27s

Confirm that all GPU nodes are labeled with clique ids:

$ (echo -e "NODE\tLABEL\tCLIQUE"; kubectl get nodes -o json | \
    jq -r '.items[] | [.metadata.name, "nvidia.com/gpu.clique", .metadata.labels["nvidia.com/gpu.clique"]] | @tsv') | \
    column -t
NODE                 LABEL                  CLIQUE
gb-nvl-043-bianca-1  nvidia.com/gpu.clique  9277d399-0674-44a9-b64e-d85bb19ce2b0.32766
gb-nvl-043-bianca-2  nvidia.com/gpu.clique  9277d399-0674-44a9-b64e-d85bb19ce2b0.32766

Simple IMEX channel injection with IMEX daemon running

Run a simple test to validate IMEX daemons are started and IMEX channels are injected:

cat <<EOF > imex-channel-injection.yaml
---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: imex-channel-injection
spec:
  numNodes: 1
  channel:
    resourceClaimTemplate:
      name: imex-channel-0
---
apiVersion: v1
kind: Pod
metadata:
  name: imex-channel-injection
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: nvidia.com/gpu.clique
            operator: Exists
  containers:
  - name: ctr
    image: ubuntu:22.04
    command: ["bash", "-c"]
    args: ["ls -la /dev/nvidia-caps-imex-channels; trap 'exit 0' TERM; sleep 9999 & wait"]
    resources:
      claims:
      - name: imex-channel-0
  resourceClaims:
  - name: imex-channel-0
    resourceClaimTemplateName: imex-channel-0
EOF

$ kubectl apply -f imex-channel-injection.yaml
computedomain.resource.nvidia.com/imex-channel-injection created
pod/imex-channel-injection created

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
imex-channel-injection   1/1     Running   0          3s

$ kubectl get pods -n nvidia-dra-driver-gpu -l resource.nvidia.com/computeDomain
NAME                                 READY   STATUS    RESTARTS   AGE
imex-channel-injection-6k9sx-ffgpf   1/1     Running   0          3s

$ kubectl logs imex-channel-injection
total 0
drwxr-xr-x 2 root root     60 Feb 19 10:43 .
drwxr-xr-x 6 root root    380 Feb 19 10:43 ..
crw-rw-rw- 1 root root 507, 0 Feb 19 10:43 channel0

$ kubectl logs -n nvidia-dra-driver-gpu -l resource.nvidia.com/computeDomain --tail=-1
/etc/nvidia-imex/nodes_config.cfg:
10.115.131.8
IMEX Log initializing at: 3/27/2025 15:47:10.092
[Mar 27 2025 15:47:10] [INFO] [tid 39] IMEX version 570.124.06 is running with the following configuration options

[Mar 27 2025 15:47:10] [INFO] [tid 39] Logging level = 4

[Mar 27 2025 15:47:10] [INFO] [tid 39] Logging file name/path = /var/log/nvidia-imex.log

[Mar 27 2025 15:47:10] [INFO] [tid 39] Append to log file = 0

[Mar 27 2025 15:47:10] [INFO] [tid 39] Max Log file size = 1024 (MBs)

[Mar 27 2025 15:47:10] [INFO] [tid 39] Use Syslog file = 0

[Mar 27 2025 15:47:10] [INFO] [tid 39] IMEX Library communication bind interface =

[Mar 27 2025 15:47:10] [INFO] [tid 39] IMEX library communication bind port = 50000

[Mar 27 2025 15:47:10] [INFO] [tid 39] Identified this node as ID 0, using bind IP of '10.115.131.8', and network interface of enP5p9s0
[Mar 27 2025 15:47:10] [INFO] [tid 39] nvidia-imex persistence file /var/run/nvidia-imex/persist.dat does not exist.  Assuming no previous importers.
[Mar 27 2025 15:47:10] [INFO] [tid 39] NvGpu Library version matched with GPU Driver version
[Mar 27 2025 15:47:10] [INFO] [tid 63] Started processing of incoming messages.
[Mar 27 2025 15:47:10] [INFO] [tid 64] Started processing of incoming messages.
[Mar 27 2025 15:47:10] [INFO] [tid 65] Started processing of incoming messages.
[Mar 27 2025 15:47:10] [INFO] [tid 39] Creating gRPC channels to all peers (nPeers = 1).
[Mar 27 2025 15:47:10] [INFO] [tid 66] Started processing of incoming messages.
[Mar 27 2025 15:47:10] [INFO] [tid 39] IMEX_WAIT_FOR_QUORUM != FULL, continuing initialization without waiting for connections to all nodes.
[Mar 27 2025 15:47:10] [INFO] [tid 67] Connection established to node 0 with ip address 10.115.131.8. Number of times connected: 1
[Mar 27 2025 15:47:10] [INFO] [tid 39] GPU event successfully subscribed

$ kubectl delete -f imex-channel-injection.yaml
computedomain.resource.nvidia.com "imex-channel-injection" deleted
pod "imex-channel-injection" deleted

Multi-Node MPI test

Install the latest version of the MPI Operator:

kubectl create -f https://github.com/kubeflow/mpi-operator/releases/download/v0.6.0/mpi-operator.yaml

Run a multi-node nvbandwidth test requiring IMEX channels with MPI:

cat <<EOF > nvbandwidth-test-job.yaml
---
apiVersion: resource.nvidia.com/v1beta1
kind: ComputeDomain
metadata:
  name: nvbandwidth-test-compute-domain
spec:
  numNodes: 2
  channel:
    resourceClaimTemplate:
      name: nvbandwidth-test-compute-domain-channel

---
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
  name: nvbandwidth-test
spec:
  slotsPerWorker: 4
  launcherCreationPolicy: WaitForWorkersReady
  runPolicy:
    cleanPodPolicy: Running
  sshAuthMountPath: /home/mpiuser/.ssh
  mpiReplicaSpecs:
    Launcher:
      replicas: 1
      template:
        metadata:
          labels:
            nvbandwidth-test-replica: mpi-launcher
        spec:
          affinity:
            nodeAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
                nodeSelectorTerms:
                - matchExpressions:
                  - key: node-role.kubernetes.io/control-plane
                    operator: Exists
          containers:
          - image: ghcr.io/nvidia/k8s-samples:nvbandwidth-v0.7-8d103163
            name: mpi-launcher
            securityContext:
              runAsUser: 1000
            command:
            - mpirun
            args:
            - --bind-to
            - core
            - --map-by
            - ppr:4:node
            - -np
            - "8"
            - --report-bindings
            - -q
            - nvbandwidth
            - -t
            - multinode_device_to_device_memcpy_read_ce
    Worker:
      replicas: 2
      template:
        metadata:
          labels:
            nvbandwidth-test-replica: mpi-worker
        spec:
          affinity:
            podAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: nvbandwidth-test-replica
                    operator: In
                    values:
                    - mpi-worker
                topologyKey: nvidia.com/gpu.clique
          containers:
          - image: ghcr.io/nvidia/k8s-samples:nvbandwidth-v0.7-8d103163
            name: mpi-worker
            securityContext:
              runAsUser: 1000
            env:
            command:
            - /usr/sbin/sshd
            args:
            - -De
            - -f
            - /home/mpiuser/.sshd_config
            resources:
              limits:
                nvidia.com/gpu: 4
              claims:
              - name: compute-domain-channel
          resourceClaims:
          - name: compute-domain-channel
            resourceClaimTemplateName: nvbandwidth-test-compute-domain-channel
EOF

$ kubectl apply -f nvbandwidth-test-job.yaml
computedomain.resource.nvidia.com/nvbandwidth-test-compute-domain created
mpijob.kubeflow.org/nvbandwidth-test created

$ kubectl get pods
NAME                              READY   STATUS      RESTARTS   AGE
nvbandwidth-test-launcher-lzv84   1/1     Running     0          3s
nvbandwidth-test-worker-0         1/1     Running     0          15s
nvbandwidth-test-worker-1         1/1     Running     0          15s

$ kubectl get pods -n nvidia-dra-driver-gpu -l resource.nvidia.com/computeDomain
NAME                                          READY   STATUS    RESTARTS   AGE
nvbandwidth-test-compute-domain-ht24d-9jhmj   1/1     Running   0          20s
nvbandwidth-test-compute-domain-ht24d-rcn2c   1/1     Running   0          20s

$ kubectl logs --tail=-1 -l job-name=nvbandwidth-test-launcher
Warning: Permanently added '[nvbandwidth-test-worker-0.nvbandwidth-test.default.svc]:2222' (ECDSA) to the list of known hosts.
Warning: Permanently added '[nvbandwidth-test-worker-1.nvbandwidth-test.default.svc]:2222' (ECDSA) to the list of known hosts.
[nvbandwidth-test-worker-0:00025] MCW rank 0 bound to socket 0[core 0[hwt 0]]: [B/././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-0:00025] MCW rank 1 bound to socket 0[core 1[hwt 0]]: [./B/./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-0:00025] MCW rank 2 bound to socket 0[core 2[hwt 0]]: [././B/././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-0:00025] MCW rank 3 bound to socket 0[core 3[hwt 0]]: [./././B/./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-1:00025] MCW rank 4 bound to socket 0[core 0[hwt 0]]: [B/././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-1:00025] MCW rank 5 bound to socket 0[core 1[hwt 0]]: [./B/./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-1:00025] MCW rank 6 bound to socket 0[core 2[hwt 0]]: [././B/././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
[nvbandwidth-test-worker-1:00025] MCW rank 7 bound to socket 0[core 3[hwt 0]]: [./././B/./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.][./././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././././.]
nvbandwidth Version: v0.7
Built from Git version: v0.7

MPI version: Open MPI v4.1.4, package: Debian OpenMPI, ident: 4.1.4, repo rev: v4.1.4, May 26, 2022
CUDA Runtime Version: 12080
CUDA Driver Version: 12080
Driver Version: 570.124.06

Process 0 (nvbandwidth-test-worker-0): device 0: HGX GB200 (00000008:01:00)
Process 1 (nvbandwidth-test-worker-0): device 1: HGX GB200 (00000009:01:00)
Process 2 (nvbandwidth-test-worker-0): device 2: HGX GB200 (00000018:01:00)
Process 3 (nvbandwidth-test-worker-0): device 3: HGX GB200 (00000019:01:00)
Process 4 (nvbandwidth-test-worker-1): device 0: HGX GB200 (00000008:01:00)
Process 5 (nvbandwidth-test-worker-1): device 1: HGX GB200 (00000009:01:00)
Process 6 (nvbandwidth-test-worker-1): device 2: HGX GB200 (00000018:01:00)
Process 7 (nvbandwidth-test-worker-1): device 3: HGX GB200 (00000019:01:00)

Running multinode_device_to_device_memcpy_read_ce.
memcpy CE GPU(row) -> GPU(column) bandwidth (GB/s)
           0         1         2         3         4         5         6         7
 0       N/A    798.02    798.25    798.02    798.02    797.88    797.73    797.95
 1    798.10       N/A    797.80    798.02    798.02    798.25    797.88    798.02
 2    797.95    797.95       N/A    797.73    797.80    797.95    797.95    797.65
 3    798.10    798.02    797.95       N/A    798.02    798.10    797.88    797.73
 4    797.80    798.02    798.02    798.02       N/A    797.95    797.80    798.02
 5    797.80    797.95    798.10    798.10    797.95       N/A    797.95    797.88
 6    797.73    797.95    798.10    798.02    797.95    797.88       N/A    797.80
 7    797.88    798.02    797.95    798.02    797.88    797.95    798.02       N/A

SUM multinode_device_to_device_memcpy_read_ce 44685.29

NOTE: The reported results may not reflect the full capabilities of the platform.
Performance can vary with software drivers, hardware clocks, and system topology.

$ kubectl delete -f nvbandwidth-test-job.yaml
computedomain.resource.nvidia.com "nvbandwidth-test-compute-domain" deleted
mpijob.kubeflow.org "nvbandwidth-test" deleted

[johnbelamaric]

@klueska is there a publicly shared version of your document describing the overall design of this feature?

[klueska]

The document you saw before is now public:
https://docs.google.com/document/d/1PrdDofsPFVJuZvcv-vtlI9n2eAh-YVf_fRQLIVmDwVY/edit?tab=t.0

[catblade]

@klueska are there instructions somewhere on how to configure the runtime?

Warning FailedCreatePodSandBox 0s (x13 over 2m33s) kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for "nvidia" is configured

[klueska]

What pod is this error coming from?

DRA doesn't require the Nvidia runtime, and for other pods, it should be configured automatically by the nvidia container toolkit installed by the GPU operator.

If you instead do a host installed toolkit (rather than a GPU Operator one), then you need to do the following to get it set up: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-containerd-for-kubernetes

[ZiMengSheng]

I see that Pod has also applied for the nvidia.com/gpu extension resource. Do we still need device-plugin now? Why?

[pickymodel]

Same question here. If DRA is enabled, do we still need to deploy device plugin?

I suppose DRA will handle the hardware device injection into the container in the Pod.

[klueska]

By default the DRA driver is only configured to enable ComputeDomains, you still need to request GPUs as you always have with the device plugin. It's possible to also enable GPU allocation from the DRA driver and then explicitly request GPUs using ResourceClaims, but this is not the default.

[ZiMengSheng]

how does
/var/lib/kubelet/plugins/compute-domain.nvidia.com/domains be used when Daemonset Pods are started, I don't see it in containerdEdits

[klueska]

The /etc/nvidia-imex folder created under /var/lib/kubelet/plugins/compute-domain.nvidia.com/domains/<computeDomainUUID>gets injected here:
https://github.com/NVIDIA/k8s-dra-driver-gpu/blob/main/cmd/compute-domain-kubelet-plugin/computedomain.go#L180-L186