NCCL NET preflight checks a driver flag that R595 removed

[yuanchen8911]

Problem

preflightGB200NetNVregFlag (validators/performance/nccl_preflight_nvreg.go) gates the nccl-all-reduce-bw-net benchmark on a node-level check: it probes every target GPU node and requires ^GrdmaPciTopoCheckOverride: 1$ in /proc/driver/nvidia/params.

That parameter does not exist in NVIDIA driver R595. Measured on p6e-gb300r.36xlarge under 595.91.07 after a genuine module unload and reload: /proc/driver/nvidia/params lists 44 parameters and zero matching grdma or topo, while CoherentGPUMemoryMode: "driver" is present.

So on any R595 cluster the preflight fails closed with a remediation hint that cannot be followed.

Why the parameter is gone

R595 is documented as backward incompatible with P6e-GB200 EFA. It drops the R580-era NVreg_GrdmaPciTopoCheckOverride escape hatch and substitutes a topology check: EFA must sit in an IOMMU group, or GPU and EFA must share a root port. R595 also introduces Coherent Driver-Based Memory Management for GB200, which is why the R595 driver container emits NVreg_CoherentGPUMemoryMode=driver as a new base directive.

The check is therefore doing the right thing for the wrong reason — it fails on R595, which is correct, but it reports a missing flag rather than an incompatible driver.

Two separate defects in the current remediation hint

nvregDocsHint tells the operator to:

1. Set the flag via spec.driver.kernelModuleConfig.name → a ConfigMap. On R595 this cannot work; the parameter does not exist and the kernel silently ignores it.
2. "then delete the nvidia-driver DaemonSet pods to pick up the change." This does not reload the module. k8s-driver-manager compares DRIVER_CONFIG_DIGEST — derived from the ClusterPolicy spec, not the ConfigMap contents — and logs "The NVIDIA driver is already loaded with the desired version and configuration, skipping the uninstallation of the driver in an attempt to not disrupt running workloads." Verified directly: a pod delete left the module untouched, and forcing a real reload required changing the ClusterPolicy spec so the digest moved.

Defect 2 is independent of the driver version and would mislead an operator on R580 too.

Proposed

Make the preflight driver-version-aware:

• R580 (and earlier) — keep the current flag check. It is correct there, and 580.173.02 is the pin AICR ships (see Audit and upgrade pinned component versions across the recipe catalog #2383).
• R595 and later — do not check for the flag. Either verify the topology property directly (EFA in an IOMMU group, or GPU and EFA sharing a root port), or report the driver as incompatible with P6e EFA and fail with that reason.
• Never silently skip. A version-aware check that skips on R595 would let validation pass while EFA quietly does not work — worse than the current failure, because it looks like success.
• Fix the remediation hint's second half regardless of version: deleting driver pods does not reload the module when the config digest is unchanged.

Reachability

Not currently reachable in a shipped configuration — #2439 holds the driver at 580.173.02 precisely because R595 breaks P6e EFA, so stock recipes stay on R580 where the check is correct. This becomes live the moment any recipe moves to R595, and it is a trap for anyone running a local build or a custom pin against a newer driver.

Out of scope

• Whether R595 can be adopted on P6e at all. Tracked in Audit and upgrade pinned component versions across the recipe catalog #2383; the answer today is no.
• Adding a native GB300 benchmark profile. The validator currently logs accelerator=gb200 ... profile=gb200/eks and overrides criteria-derived applicability on GB300, so GB300 is measured against a GB200 profile. Separate concern.
• Whether a second EFA per node would improve GB300 throughput. One EFA per node passed under R580 at 43.16 GB/s, so it is not a correctness issue.

Related: #2383 (version sweep, driver held at R580), #2439 (the bump that surfaced this)