Stop starting Fabric Manager from H100/B200 cloud-init (#552)

Fixes
https://github.com/pytorch/pytorch/actions/runs/25700981763/job/75480837337

## Summary

#541 added an eager `systemctl start nvidia-fabricmanager` and a
`modprobe nvidia-caps-imex-channels` to the H100 and B200 node-setup
scripts, both wrapped in hard-fail blocks. On AL2023's
`amazon-eks-node-al2023-x86_64-nvidia-*` AMI this fails reliably and
takes the node out of the pool before it can join the cluster. This PR
removes the FM/IMEX block from both scripts and documents why we do
not manage Fabric Manager from cloud-init.

## Root cause #1: cloud-init / FM startup race

Cloud-init runs while the GPU-resident *local FM* instances are still
initializing. The host FM completes NVSwitch routing in seconds, then
waits up to its 30 s ``GFM Wait Timeout`` for the GPU-side local-FM
instances to register over NVLink Inband. During cloud-init they are
typically not ready, so host FM exits with:

```
[ERROR] not all local fabric manager instances finished their configuration
[ERROR] fabric manager error handler is called for unhandled config error type 8
[ERROR] failed to configure all the available GPUs or NVSwitches
```

On the same node, starting FM later (via ``systemctl start`` 90 minutes
after boot, or via systemd's own normal boot path) completes in well
under one second. The AMI already runs ``systemctl enable
nvidia-fabricmanager`` during AMI build (see
``install-nvidia-driver.sh``
in awslabs/amazon-eks-ami), so systemd handles FM at a much later boot
phase than cloud-init -- which is exactly what we want.

## Root cause #2: `nvidia-caps-imex-channels` is not a kmod

On AL2023's NVIDIA AMI, the IMEX char device class (major 242) is
registered by ``nvidia.ko`` itself; there is no separate
``nvidia-caps-imex-channels.ko``. ``modprobe nvidia-caps-imex-channels``
returns ``Module not found.`` on every node booted from this AMI.

Single-node NVLS multicast on p5/p6-b200 also does not require an IMEX
channel device -- NCCL uses POSIX FD handles intranode. IMEX channels
are only needed for multi-node NVLink (MNNVL / GB200 UltraServers). The
``/etc/modules-load.d/nvidia-caps-imex-channels.conf`` write is moot.

## Verification on a live p5.48xlarge

On a node provisioned with the current (broken) script:

- ``/proc/devices`` shows ``242 nvidia-caps-imex-channels``: IMEX
  support is already in ``nvidia.ko``.
- ``systemctl start nvidia-fabricmanager`` post-boot reaches ``active
  (running)`` in <1 s and the log ends with "Successfully configured
  all the available NVSwitches to route GPU NVLink traffic" and "FM
  starting NvLink Inband".
- The originally-failed FM run during cloud-init hit the 30 s timeout
  at the exact ``GFM Wait Timeout`` value, with the same hardware and
  driver versions.

This confirms the failure is the cloud-init race, not a missing IMEX
module or a hardware issue.

## What this PR does NOT fix

- The NVLS multicast issue from #541's description ("Failed to bind
  NVLink SHARP (NVLS) Multicast memory ... CUDA error 401") is a
  separate problem that requires verifying the new node-setup actually
  lets a multi-process NCCL collective complete. With FM running
  (which the AMI now does on its own), single-node NVLS should work
  via POSIX FD handles. If a fresh ``linux.aws.h100.8`` run still
  fails, the next investigation is on the FM/driver version pin and
  systemd unit ordering, not on cloud-init.

### Testing

Clean things up manually, reboot the instances and H100 jobs are now
working again
https://github.com/pytorch/pytorch/actions/runs/25700981763/job/75482680206

## References

- Original commit: #541 (03116d1)
- AWS EKS-optimized AMI components for p5/p6:

https://docs.aws.amazon.com/eks/latest/userguide/ml-eks-optimized-ami.html
- NVIDIA confirming IMEX channels are MNNVL-only:
  NVIDIA/nccl#1632

Authored by Claude.

Author: huydhn

osdc/modules/nodepools-b200/scripts/b200-node-setup.sh

@@ -70,40 +70,27 @@ systemctl enable cpu-performance.service
 # Enable GPU persistence mode for consistent performance
 nvidia-smi -pm 1 || true
 
-# ---- Fabric Manager + IMEX channel (required for NCCL NVLS multicast) ----
-# NVSwitch-connected multi-GPU nodes need the Fabric Manager running and the
-# IMEX channel device available for NCCL to use NVLS. Without these, any
-# multi-process NCCL collective will fail with:
-#   "Failed to bind NVLink SHARP (NVLS) Multicast memory"
-cat >/etc/modules-load.d/nvidia-caps-imex-channels.conf <<'EOFS'
-nvidia-caps-imex-channels
-EOFS
-
-systemctl enable nvidia-fabricmanager || {
-  echo "ERROR: failed to enable nvidia-fabricmanager" >&2
-  exit 1
-}
-
-systemctl start nvidia-fabricmanager || {
-  systemctl status nvidia-fabricmanager --no-pager || true
-  echo "ERROR: failed to start nvidia-fabricmanager" >&2
-  exit 1
-}
-
-systemctl is-active --quiet nvidia-fabricmanager || {
-  systemctl status nvidia-fabricmanager --no-pager || true
-  echo "ERROR: nvidia-fabricmanager is not active" >&2
-  exit 1
-}
-
-modprobe nvidia-caps-imex-channels || {
-  echo "ERROR: failed to load nvidia-caps-imex-channels" >&2
-  exit 1
-}
-
-if ! lsmod | grep -q '^nvidia_caps_imex_channels '; then
-  echo "ERROR: nvidia-caps-imex-channels is not loaded" >&2
-  exit 1
-fi
+# ---- Fabric Manager / IMEX: do nothing here ----
+# Background for future maintainers:
+#  * The AMI (amazon-eks-node-al2023-x86_64-nvidia-*) already installs and
+#    enables nvidia-fabricmanager and nvidia-persistenced. systemd brings FM
+#    up later in boot, after the GPU-resident "local FM" instances have
+#    initialized.
+#  * Starting FM from cloud-init (user_data) races with that GPU-side init:
+#    the host FM finishes NVSwitch routing in seconds but then waits up to
+#    GFM Wait Timeout (30 s) for the GPU local-FM instances to register over
+#    NVLink Inband. During cloud-init they are usually not ready yet, so FM
+#    exits with "config error type 8 / not all local fabric manager
+#    instances finished their configuration", taking the whole node out.
+#  * IMEX support on this AMI is compiled into nvidia.ko (char device class
+#    nvidia-caps-imex-channels, major 242). There is no separate
+#    nvidia-caps-imex-channels.ko; `modprobe nvidia-caps-imex-channels`
+#    will always fail.
+#  * Single-node NVLS multicast on p6-b200 does not require an IMEX channel
+#    device - NCCL uses POSIX FD handles intranode. IMEX channels are only
+#    needed for multi-node NVLink (MNNVL / GB200 UltraServers).
+# If a fresh provision ever shows FM failing on the normal boot path (not
+# cloud-init), prefer a systemd drop-in with After=nvidia-persistenced.service
+# and Restart=on-failure over taking control of FM here.
 
 echo "Performance configuration complete for p6-b200.48xlarge"

osdc/modules/nodepools-h100/scripts/h100-node-setup.sh

@@ -70,40 +70,27 @@ systemctl enable cpu-performance.service
 # Enable GPU persistence mode for consistent performance
 nvidia-smi -pm 1 || true
 
-# ---- Fabric Manager + IMEX channel (required for NCCL NVLS multicast) ----
-# NVSwitch-connected multi-GPU nodes (p5.48xlarge) need the Fabric Manager
-# running and the IMEX channel device available for NCCL to use NVLS.
-# Without these, any multi-process NCCL collective will fail with:
-#   "Failed to bind NVLink SHARP (NVLS) Multicast memory"
-cat >/etc/modules-load.d/nvidia-caps-imex-channels.conf <<'EOFS'
-nvidia-caps-imex-channels
-EOFS
-
-systemctl enable nvidia-fabricmanager || {
-  echo "ERROR: failed to enable nvidia-fabricmanager" >&2
-  exit 1
-}
-
-systemctl start nvidia-fabricmanager || {
-  systemctl status nvidia-fabricmanager --no-pager || true
-  echo "ERROR: failed to start nvidia-fabricmanager" >&2
-  exit 1
-}
-
-systemctl is-active --quiet nvidia-fabricmanager || {
-  systemctl status nvidia-fabricmanager --no-pager || true
-  echo "ERROR: nvidia-fabricmanager is not active" >&2
-  exit 1
-}
-
-modprobe nvidia-caps-imex-channels || {
-  echo "ERROR: failed to load nvidia-caps-imex-channels" >&2
-  exit 1
-}
-
-if ! lsmod | grep -q '^nvidia_caps_imex_channels '; then
-  echo "ERROR: nvidia-caps-imex-channels is not loaded" >&2
-  exit 1
-fi
+# ---- Fabric Manager / IMEX: do nothing here ----
+# Background for future maintainers:
+#  * The AMI (amazon-eks-node-al2023-x86_64-nvidia-*) already installs and
+#    enables nvidia-fabricmanager and nvidia-persistenced. systemd brings FM
+#    up later in boot, after the GPU-resident "local FM" instances have
+#    initialized.
+#  * Starting FM from cloud-init (user_data) races with that GPU-side init:
+#    the host FM finishes NVSwitch routing in seconds but then waits up to
+#    GFM Wait Timeout (30 s) for the GPU local-FM instances to register over
+#    NVLink Inband. During cloud-init they are usually not ready yet, so FM
+#    exits with "config error type 8 / not all local fabric manager
+#    instances finished their configuration", taking the whole node out.
+#  * IMEX support on this AMI is compiled into nvidia.ko (char device class
+#    nvidia-caps-imex-channels, major 242). There is no separate
+#    nvidia-caps-imex-channels.ko; `modprobe nvidia-caps-imex-channels`
+#    will always fail.
+#  * Single-node NVLS multicast on p5 does not require an IMEX channel
+#    device - NCCL uses POSIX FD handles intranode. IMEX channels are only
+#    needed for multi-node NVLink (MNNVL / GB200 UltraServers).
+# If a fresh provision ever shows FM failing on the normal boot path (not
+# cloud-init), prefer a systemd drop-in with After=nvidia-persistenced.service
+# and Restart=on-failure over taking control of FM here.
 
 echo "Performance configuration complete for p5.48xlarge"