Enable CUDA fabric-handle IPC on H100 nodes and add a dedicated fabric runner#948
Open
jeanschmidt wants to merge 4 commits into
Open
Enable CUDA fabric-handle IPC on H100 nodes and add a dedicated fabric runner#948jeanschmidt wants to merge 4 commits into
jeanschmidt wants to merge 4 commits into
Conversation
…c runner **Impact:** H100 CI (p5.48xlarge node provisioning + a new `mt-l-x86iamx-44-450-h100-fab` runner) **Risk:** medium ## What Creates the IMEX channel device on H100 nodes so PyTorch expandable-segments IPC can use CUDA fabric handles, adds an optional `imex_channels` field to runner defs (renders `NVIDIA_IMEX_CHANNELS` onto the job container), a dedicated low-cap fabric runner def, and a manual validation probe to confirm fabric handles are actually used. ## Why CUDA VMM fabric handles (`CU_MEM_HANDLE_TYPE_FABRIC`), used by torch expandable-segments IPC, need `/dev/nvidia-caps-imex-channels/channel0` to exist even on a single host. The driver doesn't auto-create it, so without this the allocator silently falls back to POSIX-fd handles — a "false green" where the workload passes but never exercised the fabric path. This wires up the channel end-to-end (node device → container env → validation). ## Notes - `h100-node-setup.sh` creates the channel for the current boot and enables a `oneshot` systemd unit for reboots. Creation runs under `if` so a failure logs but never trips `set -e` — a node without fabric handles is recoverable, a failed cloud-init is not. This runs on every H100 node provision. - The runner template change is byte-identical to prior output when `imex_channels` is unset; the field gates on `is not None` so `0` is honored (covered by the new tests). - `validation/` is a **manual** probe (README + Pod + `fabric_probe.py`), intentionally not wired into `just test` or any deploy. Run it by hand (kubectl Pod or a `workflow_dispatch` job on the fabric runner) after rollout to prove fabric was used and catch silent POSIX fallback. - The new fabric runner is capped at `max_runners: 1` and depends on p5.48xlarge Capacity Blocks being available. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
tofu plan — meta-prod-aws-ue2✅ Plan succeeded · commit Plan output |
- Add docs/h100-fabric-handles-imex-channels.md covering CUDA CU_MEM_HANDLE_TYPE_FABRIC support on the OSDC H100 fleet - Record live-node verification (p5.48xlarge, 2026-07-24): fabric allocate/export/import round-trip passes; only the IMEX channel device is missing, a per-node config gap, not a hardware gap - Detail the fabric-enablement design: node-side mknod channel creation + unprivileged pod injection via NVIDIA_IMEX_CHANNELS, gated per-def to H100, with fails-closed ordering - Correct the Fabric Manager / IMEX race notes and the "multi-node only" script comment for the CUDA fabric-handle path - Link the new doc from the CLAUDE.md docs index Notes: Prompted by pytorch/pytorch#190860 (fabric-handle destructor crash). Debunks the claim that AWS p5 reports fabric NOT_SUPPORTED and that fabric CI needs GB200 hardware or the nvidia-imex daemon. Documentation only — no infrastructure or runner behavior changes in this commit. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
tofu plan — meta-prod-aws-uw1✅ Plan succeeded · commit Plan output |
tofu plan — meta-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue1✅ Plan succeeded · commit Plan output |
tofu plan — lf-prod-aws-ue2✅ Plan succeeded · commit Plan output |
- Add 1/2/4/8-GPU fabric runner defs (22-225, 44-450-2, 88-900-4, 176-1800-8) - Replace the single dedicated 2-GPU fabric runner (max_runners: 1) with the sized family - Give the 8-GPU full-node runner its own p5-large fleet to avoid single-numa-node TopologyAffinityError - Set per-def max_runners with a meta-prod-aws-uw1 override sized to the reserved 8-GPU node capacity - Remove the manual fabric-IPC validation probe (README, pod manifest, fabric_probe.py) Notes: The prior fabric rollout shipped one low-cap 2-GPU runner plus a manual probe to guard against silent POSIX-fd fallback. This generalizes fabric access to every GPU shape so jobs can request 1, 2, 4, or 8 H100s, with caps derived from the fixed reservation (8 GPUs/node): 8/4/2/1 concurrent runners by GPU count, scaled up on meta-prod-aws-uw1. The 8-GPU pod cannot satisfy the shared p5-48xlarge pool's single-numa-node policy across two NUMA nodes, so it runs on a dedicated p5-large fleet. The validation probe was a one-off manual check and is dropped now that fabric is standard across the fleet. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
Contributor
Author
|
This change node startup script osdc/modules/nodepools-h100/scripts/h100-node-setup.sh My main concern is that this can introduce failure modes that will make nodes fail to startup. Please review very carefully it. Is it written defensively? Does it have retries in place? If it fails, can this blows up the instance startup preventing it from joining? In case of failures, are there fallbacks and protections? |
|
Claude finished @jeanschmidt's task in 1m 55s —— View job Review:
|
| Question | Answer |
|---|---|
| Written defensively? | Yes for the channel creation (guarded if !, idempotent, pre-checks /proc/devices). Partially for the surrounding plumbing (heredoc write + 2× systemctl run under bare set -e). |
| Retries in place? | No. IMDS has retries; the IMEX helper does not. This is the one gap I'd fix (#1). |
| Can a failure block node join? | Not from the channel creation itself — it's fail-soft by design. The only hard-fail surfaces are the heredoc write and systemctl daemon-reload/enable, which are low-risk and match the existing cpu-performance pattern. |
| Fallbacks / protections? | Yes — functional fallback (POSIX-fd handles) when the channel is absent, and the reboot oneshot re-creates it. The gap is that the oneshot doesn't cover the first boot (#1). |
Bottom line: The change is safe with respect to your primary worry — it won't take a node out. The one thing I'd genuinely recommend before merge is a bounded retry in the helper (#1), because without it the first-boot driver-not-ready race reintroduces the exact "false green" the PR sets out to kill. #2 is optional hardening.
· branch osdc-h100-imex-fabric-channels
- Poll /proc/devices for nvidia-caps-imex-channels up to 5 times, 2s apart - Break as soon as the char major appears; log each retry attempt to stderr - Update the absent-class error to note retries were exhausted nvidia.ko may not have registered the nvidia-caps-imex-channels char class in /proc/devices by the time the IMEX helper first runs at boot, causing the channel node creation to fail spuriously. Retrying gives the driver time to register the dynamically-assigned major before giving up. Signed-off-by: Jean Schmidt <contato@jschmidt.me>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Impact: H100 nodes
Risk: medium
What
Creates the IMEX channel device on H100 nodes so PyTorch expandable-segments IPC can use CUDA fabric handles, adds an optional
imex_channelsfield to runner defs (rendersNVIDIA_IMEX_CHANNELSonto the job container), a dedicated low-cap fabric runner def, and a manual validation probe to confirm fabric handles are actually used.Why
CUDA VMM fabric handles (
CU_MEM_HANDLE_TYPE_FABRIC), used by torch expandable-segments IPC, need/dev/nvidia-caps-imex-channels/channel0to exist even on a single host. The driver doesn't auto-create it, so without this the allocator silently falls back to POSIX-fd handles — a "false green" where the workload passes but never exercised the fabric path. This wires up the channel end-to-end (node device → container env → validation).Notes
h100-node-setup.shcreates the channel for the current boot and enables aoneshotsystemd unit for reboots. Creation runs underifso a failure logs but never tripsset -e— a node without fabric handles is recoverable, a failed cloud-init is not. This runs on every H100 node provision.imex_channelsis unset; the field gates onis not Noneso0is honored (covered by the new tests).max_runners: 1and depends on p5.48xlarge Capacity Blocks being available.