Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions .github/workflows/nvml-mock-e2e.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,22 @@ on:
type: string

jobs:
mockib-integration:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Install infiniband-diags
run: sudo apt-get update && sudo apt-get install -y infiniband-diags

- name: Build libibmocksys and run integration tests
run: |
make -C pkg/network/mockibsysfs
go test ./pkg/network/mockibsysfs/...
go test -tags=integration ./pkg/network/mockibsysfs/render/...

e2e-device-plugin:
needs: mockib-integration
runs-on: ubuntu-latest
strategy:
fail-fast: false
Expand Down Expand Up @@ -141,6 +156,13 @@ jobs:
# IB-enabled profiles are expected to expose 2 HCAs.
./tests/e2e/validate-ibstat.sh "$POD" "${{ matrix.gpu-profile }}" 2

- name: Validate ibping (vendor MAD / libibumad)
run: |
chmod +x tests/e2e/validate-ibping.sh
POD=$(kubectl get pods -l app.kubernetes.io/name=nvml-mock \
-o jsonpath='{.items[0].metadata.name}')
./tests/e2e/validate-ibping.sh "$POD" "${{ matrix.gpu-profile }}" 2

- name: Deploy NVIDIA device plugin
run: |
kubectl apply -f tests/e2e/device-plugin-mock.yaml
Expand Down Expand Up @@ -780,6 +802,13 @@ jobs:
-o jsonpath='{.items[0].metadata.name}')
./tests/e2e/validate-ibstat.sh "$T4_POD" t4 2

- name: Validate ibping on A100 worker
run: |
chmod +x tests/e2e/validate-ibping.sh
A100_POD=$(kubectl get pods -l app.kubernetes.io/instance=nvml-mock-a100 \
-o jsonpath='{.items[0].metadata.name}')
./tests/e2e/validate-ibping.sh "$A100_POD" a100 2

- name: Deploy NVIDIA device plugin
run: |
kubectl apply -f tests/e2e/device-plugin-mock.yaml
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@

# Agentic working artifacts
.agents/
.cursor/
.worktrees/
AGENTS.md
CLAUDE.md
docs/plans/
docs/superpowers/
19 changes: 11 additions & 8 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]

### Added
- InfiniBand mock: `ibping` (OpenIB vendor ping MAD class only) via synthetic
umad fds, global `umad-bus/` for cross-HCA server/client, TID-matched replies,
`REGISTER_AGENT2`, and mixed-fd `poll()` with libc.
- InfiniBand mock: real `ibstat`, `ibstatus`, `iblinkinfo`, and other
`infiniband-diags` / `rdma-core` tools now work inside the nvml-mock
DaemonSet without IB hardware. Implementation: `LD_PRELOAD` shim
(`libibmocksys.so`) redirects libc filesystem accesses against
`/sys/class/infiniband*` and `/dev/infiniband` to a fake tree rendered
at startup by `render-ib-sysfs` from each profile's new `infiniband:`
block. Defaults: `a100` -> ConnectX-6 HDR; `h100` / `b200` / `gb200`
-> ConnectX-7 NDR; `l40s` / `t4` -> disabled.
- New GPU profile `gb300` modeling the NVIDIA GB300 NVL (Grace-Blackwell
Ultra Superchip): 8 GPUs/node arranged as 4 Grace+2×B300 trays, 288 GiB
HBM3e per GPU, 1.4 kW default TDP / 1.6 kW boost, PCIe Gen6, NVLink v5
Expand Down Expand Up @@ -47,14 +58,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the same string verbatim in `nvmlPciInfo.busId`; the new format aligns
the mock with what real Linux PCI sysfs exposes and is a hard
prerequisite for the PCIe sysfs renderer above. (#263)
- InfiniBand mock: real `ibstat`, `ibstatus`, `iblinkinfo`, and other
`infiniband-diags` / `rdma-core` tools now work inside the nvml-mock
DaemonSet without IB hardware. Implementation: `LD_PRELOAD` shim
(`libibmocksys.so`) redirects libc filesystem accesses against
`/sys/class/infiniband*` and `/dev/infiniband` to a fake tree rendered
at startup by `render-ib-sysfs` from each profile's new `infiniband:`
block. Defaults: `a100` -> ConnectX-6 HDR; `h100` / `b200` / `gb200`
-> ConnectX-7 NDR; `l40s` / `t4` -> disabled.

## [0.1.0] - 2026-04-07

Expand Down
15 changes: 14 additions & 1 deletion deployments/nvml-mock/helm/nvml-mock/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Deploys a DaemonSet that creates on every node:
- Node label `nvidia.com/gpu.present=true`
- A fake InfiniBand sysfs tree at `/var/lib/nvml-mock/ib/sys/class/infiniband/...`
paired with `libibmocksys.so` (`LD_PRELOAD`) so real `ibstat`, `ibstatus`,
`iblinkinfo`, ... read mock HCAs
`iblinkinfo`, `ibping`, ... use mock HCAs (sysfs + synthetic umad)
- A fake PCI sysfs tree at `/var/lib/nvml-mock/sys/bus/pci/devices/...` (symlinks
into `/var/lib/nvml-mock/sys/devices/pciDDDD:BB/...`) so topology-aware
consumers (NVIDIA DRA driver `dra.k8s.io/pcieRoot`, NUMA-aware schedulers)
Expand Down Expand Up @@ -503,8 +503,21 @@ read from the rendered tree:
POD=$(kubectl get pods -l app.kubernetes.io/name=nvml-mock -o jsonpath='{.items[0].metadata.name}')
kubectl exec "$POD" -- ibstat
kubectl exec "$POD" -- ibstatus
kubectl exec "$POD" -- ibping -c 1 -C mlx5_0 -P 1 1
```

**Environment (DaemonSet):**

| Variable | Value | Purpose |
|----------|--------|---------|
| `LD_PRELOAD` | `/usr/local/lib/libibmocksys.so.1` | Sysfs redirect + synthetic umad |
| `MOCK_IB_ROOT` | `/var/lib/nvml-mock/ib` | Fake sysfs + `umad-bus/` runtime dir |
| `MOCK_IB_DISABLE` | unset (set `1` to bypass) | Debug escape hatch |

**Limitations:** sysfs tools (`ibstat`, …) read the rendered tree. **`ibping`**
uses emulated umad (vendor ping MADs only), not real HCAs or RDMA. Runtime
MAD relay files live under `/var/lib/nvml-mock/ib/umad-bus/`.

### Defaults per profile

| Profile | Enabled | HCA | Speed | HCAs per GPU |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -79,9 +79,8 @@ spec:
{{- end }}
# LD_PRELOAD installs the InfiniBand sysfs redirector in every
# process spawned in this container (e.g. `kubectl exec ibstat`).
# The shim only rewrites paths under /sys/class/infiniband*,
# /sys/class/infiniband_mad/, /sys/class/infiniband_verbs/ and
# /dev/infiniband — every other path passes through unchanged.
# The shim rewrites IB sysfs/dev paths and emulates umad for ibping.
# Every other path passes through unchanged.
- name: LD_PRELOAD
value: "/usr/local/lib/libibmocksys.so.1"
- name: MOCK_IB_ROOT
Expand Down
10 changes: 5 additions & 5 deletions pkg/network/mockibsysfs/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,20 @@
# SPDX-License-Identifier: Apache-2.0

CC ?= gcc
CFLAGS ?= -O2 -Wall -Wextra -fPIC -fvisibility=default
LDFLAGS ?= -shared -Wl,-soname,libibmocksys.so.1 -ldl
CFLAGS ?= -O2 -Wall -Wextra -Wno-format-truncation -fPIC -fvisibility=default
LDFLAGS ?= -shared -Wl,-soname,libibmocksys.so.1 -ldl -lpthread

LIB_NAME := libibmocksys.so
LIB_SONAME := $(LIB_NAME).1
LIB_VERSION ?= 1.0.0
LIB_VERSION ?= 1.2.0
LIB_REAL := $(LIB_NAME).$(LIB_VERSION)

.PHONY: all clean

all: $(LIB_NAME)

$(LIB_REAL): shim.c
$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $<
$(LIB_REAL): shim.c umad_mock.c mock_ib_root.c umad_mock.h mock_ib_root.h
$(CC) $(CFLAGS) $(LDFLAGS) -o $@ shim.c umad_mock.c mock_ib_root.c

$(LIB_NAME): $(LIB_REAL)
ln -sf $(LIB_REAL) $(LIB_SONAME)
Expand Down
Loading
Loading