Skip to content

[Feature]: Add a configurable --sysfs-root for device NUMA detection (GetNumaNode) #1833

Description

@eliranw

Summary

GetNumaNode reads a device's NUMA node from a hardcoded /sys/bus/pci/devices/<busID>/numa_node, with no way to redirect the sysfs root.
This is the device-plugin counterpart of NVIDIA/k8s-test-infra#264 (same --sysfs-root request for the DRA driver); both consume the synthetic tree from #263 (render-pci-sysfs).

Problem

internal/rm/nvml_devices.go:150-162:

busID := strings.ToLower(strings.TrimPrefix(uint8Slice(info.BusId[:]).String(), "0000"))
b, err := os.ReadFile(fmt.Sprintf("/sys/bus/pci/devices/%s/numa_node", busID))
if err != nil {
    return false, 0, nil  // error swallowed -> device reported with no NUMA
}

The /sys prefix is a literal; nothing overrides it (--nvidia-driver-root / --nvidia-dev-root feed only CDI / library resolution, not this read). When NVML reports devices whose bus IDs are not backed by host sysfs (simulated/mock GPUs), the read fails, hasNuma=false, and BuildDevice (internal/rm/devices.go:93-101) leaves Device.Topology nil, so the device is advertised with no NUMA and the Topology Manager / podresources see no affinity.

Proposed solution

Flag --sysfs-root (default /sys), matching #264 and the existing --nvidia-*-root family:

os.ReadFile(filepath.Join(sysfsRoot, "bus/pci/devices", busID, "numa_node"))

In-tree precedent for the same read: go-nvlib's WithPCIDevicesRoot (pkg/nvpci/nvpci.go:187,317, mock at mock.go:104) and k8s dynamic-resource-allocation's WithFSFromRoot (which #264 builds on). Default unchanged means zero impact on real deployments.

Alternatives

  • nvmlDeviceGetNumaNodeId: doesn't help. A mock NVML returns NVML_ERROR_NOT_SUPPORTED for it, and on real hardware it signals "no NUMA" via return code (not the sysfs -1) and is absent on older drivers, so it would need return-code gating plus a sysfs fallback anyway.
  • Subpath-mount the tree over /sys/bus/pci/devices: possible (Add GOLANG_ARCH arg to Ubuntu 20.04 Dockerfile #264's fallback) but shadows the real PCI dir per-pod; a flag is cleaner.

Scoped to GetNumaNode (NVML full GPU + MIG, which delegates to the parent). A second hardcoded root at internal/vgpu/pciutil.go:42 (GFD vGPU detection) could adopt the same flag later. Happy to send a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions