From 2e308909f1b4de0773aa6df3eb2fed844c16a30a Mon Sep 17 00:00:00 2001 From: Jean Schmidt Date: Fri, 24 Jul 2026 15:50:44 -0700 Subject: [PATCH 1/4] Enable CUDA fabric-handle IPC on H100 nodes and add a dedicated fabric runner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **Impact:** H100 CI (p5.48xlarge node provisioning + a new `mt-l-x86iamx-44-450-h100-fab` runner) **Risk:** medium ## What Creates the IMEX channel device on H100 nodes so PyTorch expandable-segments IPC can use CUDA fabric handles, adds an optional `imex_channels` field to runner defs (renders `NVIDIA_IMEX_CHANNELS` onto the job container), a dedicated low-cap fabric runner def, and a manual validation probe to confirm fabric handles are actually used. ## Why CUDA VMM fabric handles (`CU_MEM_HANDLE_TYPE_FABRIC`), used by torch expandable-segments IPC, need `/dev/nvidia-caps-imex-channels/channel0` to exist even on a single host. The driver doesn't auto-create it, so without this the allocator silently falls back to POSIX-fd handles — a "false green" where the workload passes but never exercised the fabric path. This wires up the channel end-to-end (node device → container env → validation). ## Notes - `h100-node-setup.sh` creates the channel for the current boot and enables a `oneshot` systemd unit for reboots. Creation runs under `if` so a failure logs but never trips `set -e` — a node without fabric handles is recoverable, a failed cloud-init is not. This runs on every H100 node provision. - The runner template change is byte-identical to prior output when `imex_channels` is unset; the field gates on `is not None` so `0` is honored (covered by the new tests). - `validation/` is a **manual** probe (README + Pod + `fabric_probe.py`), intentionally not wired into `just test` or any deploy. Run it by hand (kubectl Pod or a `workflow_dispatch` job on the fabric runner) after rollout to prove fabric was used and catch silent POSIX fallback. - The new fabric runner is capped at `max_runners: 1` and depends on p5.48xlarge Capacity Blocks being available. Signed-off-by: Jean Schmidt --- .../defs/l-x86iamx-44-450-h100-fab.yaml | 12 + .../arc-runners-h100/validation/README.md | 137 ++++++++ .../validation/fabric-validation-pod.yaml | 101 ++++++ .../validation/fabric_probe.py | 293 ++++++++++++++++++ .../scripts/python/generate_runners.py | 10 + .../scripts/python/test_generate_runners.py | 46 +++ .../arc-runners/templates/runner.yaml.tpl | 2 +- .../nodepools-h100/scripts/h100-node-setup.sh | 90 +++++- 8 files changed, 678 insertions(+), 13 deletions(-) create mode 100644 osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab.yaml create mode 100644 osdc/modules/arc-runners-h100/validation/README.md create mode 100644 osdc/modules/arc-runners-h100/validation/fabric-validation-pod.yaml create mode 100644 osdc/modules/arc-runners-h100/validation/fabric_probe.py diff --git a/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab.yaml b/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab.yaml new file mode 100644 index 00000000..ec50e672 --- /dev/null +++ b/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab.yaml @@ -0,0 +1,12 @@ +# ARC runner definition: l-x86iamx-44-450-h100-fab (2x NVIDIA H100, IMEX fabric) +# Scheduled on NodePool: p5-48xlarge (from nodepools-h100 module) +runner: + name: l-x86iamx-44-450-h100-fab + instance_type: p5.48xlarge + disk_size: 200 + vcpu: 44 + memory: 450Gi + gpu: 2 + imex_channels: 0 + # Dedicated fabric runner: fixed low cap, not scaled to the full reservation. + max_runners: 1 diff --git a/osdc/modules/arc-runners-h100/validation/README.md b/osdc/modules/arc-runners-h100/validation/README.md new file mode 100644 index 00000000..76c49f95 --- /dev/null +++ b/osdc/modules/arc-runners-h100/validation/README.md @@ -0,0 +1,137 @@ +# H100 CUDA fabric-handle IPC validation probe + +Rollout validation for the H100 fabric runner (`mt-l-x86iamx-44-450-h100-fab`) +and its IMEX channel. This is a manual probe, not a unit test. It is not wired +into `just test` or any module deploy. + +## What it validates + +That a torch workload on a fabric-enabled H100 runner actually uses CUDA +**fabric handles** for expandable-segments IPC, instead of silently falling back +to POSIX file-descriptor handles. + +## Why it exists (the false-green risk) + +`c10::cuda::get_fabric_access()` decides whether the CUDA caching allocator can +share expandable memory segments over a FABRIC handle. It probes NVML and then +runs a full CUDA allocate / export / import cycle over the fabric handle +(`isFabricSupported()` in `c10/cuda/PeerToPeerAccess.cpp`). That import is what +needs the IMEX channel device inside the container. + +If the IMEX channel is missing or unusable, the cycle fails and +`ExpandableSegment::detectHandleType()` (in `c10/cuda/CUDACachingAllocator.cpp`) +GRACEFULLY falls back to POSIX-fd handles. The workload still completes and exits +0. So a green job does not prove fabric was used - it can be a silent POSIX +fallback. This is the "false green" the rollout must guard against. + +There is no Python API that reports the handle type actually used, so the probe +captures the C++ INFO logs c10 emits and asserts on them: + +| C++ log line (INFO) | Meaning | +| ----------------------------------------------------- | ----------------------------- | +| `use fabric handle to share expandable segments` | fabric export was used (good) | +| `use posix fd to share expandable segments` | silent POSIX fallback (bad) | +| `use fabric handle to import expandable segments` | fabric import was used (good) | +| `using fabric to exchange memory handles` | `isFabricSupported()` passed | +| `... falling back to fd handle exchange` | fabric probe failed | + +The probe forces the expandable-segments IPC path (allocate a >2 MiB tensor, +`reduce_tensor` to export and cache the handle, then free it to run the segment +destructor - the path PR #190860 fixed), captures the logs, and prints: + +- `FABRIC_VALIDATION: PASS` - fabric SHARE line present, no POSIX-fallback line. +- `FABRIC_VALIDATION: FAIL` - fell back to POSIX, or the path was never exercised + (inconclusive), or the workload errored. Exit code is non-zero on FAIL. + +It deliberately leaves `expandable_segments_handle_type` UNSPECIFIED. UNSPECIFIED +is the auto-detect-or-fall-back path we must validate; forcing FABRIC would make +c10 hard-error instead of falling back, hiding the exact false-green under test. + +## How it works + +`fabric_probe.py` runs the torch workload in a child process and evaluates that +child's captured C++ logs after it exits (so glog buffers are flushed). + +- Multiprocess mode (default when >=2 GPUs) mirrors + `test/distributed/test_p2p_ipc.py::test_p2p_ipc_expandable_segments`: rank 0 on + GPU 0 allocates and exports; rank 1 on GPU 1 imports without pre-allocating + (the `#179220` regression shape). This exercises the real cross-GPU fabric + round trip the IMEX channel enables, asserting both the fabric SHARE and the + fabric IMPORT log lines. +- Single-process mode (`--single-process`, or automatic on a 1-GPU host) does + allocate / export / free in one process. Because `isFabricSupported()` already + runs a self-contained export+import cycle through the IMEX channel while + deciding the handle type, one GPU is enough to prove the channel is usable and + that the SHARE path took fabric. + +Required environment (set by the manifest, or `os.environ.setdefault` when run +standalone): `NVIDIA_IMEX_CHANNELS=0`, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, +`TORCH_CUDA_EXPANDABLE_SEGMENTS_IPC=1`, `TORCH_CPP_LOG_LEVEL=INFO`, `GLOG_logtostderr=1`. + +## Image requirement + +The probe only validates the torch build it runs inside. That build MUST contain +the c10 expandable-segments fabric IPC code and be built with CUDA >= 12.4. +Prefer the exact image the H100 CI jobs run. If the build lacks the code, the +probe reports an inconclusive FAIL (no share log at all) rather than a false +pass - so a wrong image cannot masquerade as success. + +## Run it: (a) directly via kubectl on meta-prod-aws-uw1 + +From the `osdc/` repo root, against the `meta-prod-aws-uw1` cluster context: + +```bash +# 1. Edit fabric-validation-pod.yaml and set spec.containers[].image to the +# torch/CUDA image under validation (replaces REPLACE_WITH_TORCH_CUDA_IMAGE). + +# 2. Publish the probe source as a ConfigMap (single source of truth - the .py). +kubectl create configmap fabric-probe -n default \ + --from-file=fabric_probe.py=modules/arc-runners-h100/validation/fabric_probe.py + +# 3. Launch the probe Pod. +kubectl apply -f modules/arc-runners-h100/validation/fabric-validation-pod.yaml + +# 4. Watch the verdict (the last log line is FABRIC_VALIDATION: PASS/FAIL). +kubectl logs -f pod/fabric-validation -n default + +# 5. Clean up. +kubectl delete pod/fabric-validation configmap/fabric-probe -n default +``` + +A p5.48xlarge node must be available (Capacity Blocks reserved). The Pod +tolerates the H100 fleet taints and requests `nvidia.com/gpu: 2`. + +## Run it: (b) as a workflow_dispatch job on the fabric runner + +Running on the runner uses the CI job's own torch image, so it validates exactly +what production runs. No image placeholder to fill in. Example workflow: + +```yaml +name: h100-fabric-validation +on: + workflow_dispatch: +jobs: + fabric-probe: + runs-on: mt-l-x86iamx-44-450-h100-fab + steps: + - name: Checkout osdc (for the probe source) + uses: actions/checkout@v4 + - name: Run fabric validation probe + env: + NVIDIA_IMEX_CHANNELS: "0" + PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True" + TORCH_CUDA_EXPANDABLE_SEGMENTS_IPC: "1" + TORCH_CPP_LOG_LEVEL: "INFO" + GLOG_logtostderr: "1" + run: python modules/arc-runners-h100/validation/fabric_probe.py +``` + +The step fails (non-zero exit) if fabric was not used, turning the false-green +into a real red. Adjust the checkout to wherever the probe source is available +on the runner. + +## Files + +- `fabric_probe.py` - the probe (driver + torch workload + log assertion). +- `fabric-validation-pod.yaml` - the standalone kubectl Pod. +- `README.md` - this file. diff --git a/osdc/modules/arc-runners-h100/validation/fabric-validation-pod.yaml b/osdc/modules/arc-runners-h100/validation/fabric-validation-pod.yaml new file mode 100644 index 00000000..d3ec40de --- /dev/null +++ b/osdc/modules/arc-runners-h100/validation/fabric-validation-pod.yaml @@ -0,0 +1,101 @@ +# CUDA fabric-handle IPC validation Pod for the H100 fabric runner rollout. +# +# This is a manual, run-to-completion validation probe. It is NOT part of any +# module deploy and is NOT wired into `just`. Apply it by hand on the target +# cluster (meta-prod-aws-uw1), read the logs for the PASS/FAIL verdict, then +# delete it. See README.md in this directory. +# +# The probe source (fabric_probe.py) is mounted from a ConfigMap so it lives in +# exactly one place. Create the ConfigMap from the .py file before applying: +# +# kubectl create configmap fabric-probe -n default \ +# --from-file=fabric_probe.py=modules/arc-runners-h100/validation/fabric_probe.py +# kubectl apply -f modules/arc-runners-h100/validation/fabric-validation-pod.yaml +# kubectl logs -f pod/fabric-validation -n default +# kubectl delete pod/fabric-validation configmap/fabric-probe -n default +# +# IMAGE: set spec.containers[].image to the torch/CUDA build under validation. +# It MUST contain the c10 expandable-segments fabric IPC code and be built with +# CUDA >= 12.4. Prefer the exact image the H100 CI jobs run. If the build lacks +# the code the probe reports an inconclusive FAIL (no share log) rather than a +# false pass. See README.md. +apiVersion: v1 +kind: Pod +metadata: + name: fabric-validation + namespace: default + labels: + osdc.io/module: arc-runners-h100 + osdc.io/purpose: fabric-ipc-validation + annotations: + # Never let Karpenter disrupt the node mid-probe. + karpenter.sh/do-not-disrupt: "true" +spec: + restartPolicy: Never + # Pin to the p5.48xlarge H100 fleet (the fabric runner's node type). + nodeSelector: + node.kubernetes.io/instance-type: p5.48xlarge + # Tolerate every taint an H100 workflow node can carry so the probe schedules + # exactly where the fabric runner's job pods do. + tolerations: + - key: node-fleet + operator: Exists + effect: NoSchedule + - key: instance-type + operator: Exists + effect: NoSchedule + - key: nvidia.com/gpu + operator: Exists + effect: NoSchedule + - key: git-cache-not-ready + operator: Exists + effect: NoSchedule + containers: + - name: fabric-probe + # REPLACE with the torch/CUDA image under validation (see header + README). + image: REPLACE_WITH_TORCH_CUDA_IMAGE + command: ["python", "/probe/fabric_probe.py"] + env: + # Expose IMEX channel 0 to the container so cuMem*ShareableHandle with + # CU_MEM_HANDLE_TYPE_FABRIC can succeed. Without this the allocator + # silently falls back to POSIX fd and the probe fails. + - name: NVIDIA_IMEX_CHANNELS + value: "0" + # Drive the c10 allocator down the expandable-segments IPC path and turn + # on the C++ INFO logs the probe asserts against. Leave the handle type + # UNSPECIFIED so the auto-detect / fallback path is what gets validated. + - name: PYTORCH_CUDA_ALLOC_CONF + value: "expandable_segments:True" + - name: TORCH_CUDA_EXPANDABLE_SEGMENTS_IPC + value: "1" + - name: TORCH_CPP_LOG_LEVEL + value: "INFO" + - name: GLOG_logtostderr + value: "1" + resources: + requests: + cpu: "4" + memory: 32Gi + nvidia.com/gpu: "2" + limits: + cpu: "4" + memory: 32Gi + nvidia.com/gpu: "2" + volumeMounts: + - name: probe + mountPath: /probe + readOnly: true + # NCCL/torch multiprocessing want more than the default 64Mi /dev/shm. + - name: dshm + mountPath: /dev/shm + volumes: + - name: probe + configMap: + name: fabric-probe + items: + - key: fabric_probe.py + path: fabric_probe.py + - name: dshm + emptyDir: + medium: Memory + sizeLimit: 2Gi diff --git a/osdc/modules/arc-runners-h100/validation/fabric_probe.py b/osdc/modules/arc-runners-h100/validation/fabric_probe.py new file mode 100644 index 00000000..a4915bf7 --- /dev/null +++ b/osdc/modules/arc-runners-h100/validation/fabric_probe.py @@ -0,0 +1,293 @@ +#!/usr/bin/env python3 +"""Rollout validation probe for CUDA fabric-handle IPC on H100 with an IMEX channel. + +This is NOT a unit test. It is a manual validation probe for the H100 fabric +runner rollout. It must never be wired into `just test`. + +Why this probe exists +--------------------- +`c10::cuda::get_fabric_access()` (c10/cuda/PeerToPeerAccess.cpp) probes NVML plus +a full CUDA allocate/export/import cycle over a FABRIC handle. When the IMEX +channel is missing or unusable that cycle fails and the caching allocator +(`ExpandableSegment::detectHandleType`, c10/cuda/CUDACachingAllocator.cpp) +GRACEFULLY falls back to POSIX-fd handles. A torch workload therefore exits 0 +whether or not fabric handles were ever used. Exit code alone cannot tell a +fabric run apart from a silent POSIX fallback ("false green"). + +There is no Python API that reports which handle type the allocator picked, so +this probe captures the C++ INFO logs emitted by c10 and ASSERTS on them: + + - "use fabric handle to share expandable segments" -> fabric export was used + - "use posix fd to share expandable segments" -> silent POSIX fallback + - "use fabric handle to import expandable segments" -> fabric import was used + +The probe forces the allocator down the expandable-segments IPC path (allocate +>2 MiB, share via reduce_tensor to cache the exported handle, then free to run +the segment destructor), captures the logs, and fails hard unless the fabric +SHARE line is present and no POSIX-fallback line is. + +Handle-type note: the probe deliberately leaves +`expandable_segments_handle_type` UNSPECIFIED (never forces FABRIC). UNSPECIFIED +is exactly the auto-detect-or-fall-back path we must validate; forcing FABRIC +would make c10 hard-error instead of falling back, which would hide the very +false-green this probe guards against. + +Modes +----- +- multiprocess (default when >=2 GPUs): mirrors test/distributed/test_p2p_ipc.py + test_p2p_ipc_expandable_segments. Rank 0 (GPU 0) allocates and exports; rank 1 + (GPU 1) imports without pre-allocating (the #179220 regression shape). This + exercises the real cross-GPU fabric round trip the IMEX channel enables. +- single-process (fallback / 1-GPU): allocate, reduce_tensor to export, free. + isFabricSupported() already runs a self-contained export+import cycle through + the IMEX channel while deciding the handle type, so even one GPU proves the + channel is usable and that the SHARE path took fabric. Use --single-process to + force it. + +The actual torch workload runs in a child process; its C++ logs are captured +after it fully exits (so glog buffers are flushed) and then evaluated. +""" + +from __future__ import annotations + +import argparse +import os +import subprocess +import sys + +# Set before any torch import so the CUDA allocator picks these up on first +# touch. setdefault lets the k8s manifest or an operator override any of them. +_ENV_DEFAULTS = { + "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True", + "TORCH_CUDA_EXPANDABLE_SEGMENTS_IPC": "1", + "TORCH_CPP_LOG_LEVEL": "INFO", + "GLOG_logtostderr": "1", +} +for _k, _v in _ENV_DEFAULTS.items(): + os.environ.setdefault(_k, _v) + +# Contract with c10 source. These exact strings are emitted by +# c10/cuda/CUDACachingAllocator.cpp and c10/cuda/PeerToPeerAccess.cpp. +LOG_FABRIC_SHARE = "use fabric handle to share expandable segments" +LOG_POSIX_SHARE = "use posix fd to share expandable segments" +LOG_FABRIC_IMPORT = "use fabric handle to import expandable segments" +LOG_POSIX_IMPORT = "use posix fd to import expandable segments" +LOG_FABRIC_OK = "using fabric to exchange memory handles" +LOG_FABRIC_FALLBACK = "falling back to fd handle exchange" + +# float32 * 2Mi elements = 8 MiB; comfortably above the 2 MiB segment size so an +# expandable segment is created (mirrors test_p2p_ipc_expandable_segments). +TENSOR_ELEMENTS = 2 * 1024 * 1024 +MASTER_ADDR = "127.0.0.1" +MASTER_PORT = "29511" + +MODE_MARKER = "PROBE_MODE:" +VERDICT_MARKER = "FABRIC_VALIDATION:" + + +def _log(msg: str) -> None: + print(f"[probe] {msg}", flush=True) + + +def _single_process_workload() -> None: + import gc + + import torch + from torch.multiprocessing.reductions import reduce_tensor + + print(f"{MODE_MARKER} single-process", flush=True) + if not torch.cuda.is_available(): + raise SystemExit("CUDA is not available; cannot run the fabric probe") + + torch.cuda.set_device(0) + torch.cuda.memory._set_allocator_settings("expandable_segments:True") + torch.cuda.empty_cache() + + device = torch.device("cuda", 0) + mib = TENSOR_ELEMENTS * 4 / 1024 / 1024 + _log(f"allocating {mib:.0f} MiB expandable segment on {device}") + tensor = torch.randn(TENSOR_ELEMENTS, device=device) + + # reduce_tensor exports and caches the shareable handle, emitting the + # SHARE log line for the handle type the allocator selected. + meta = reduce_tensor(tensor) + _log("reduce_tensor() completed (handle exported)") + + # Free to run the ExpandableSegment destructor against the cached handle + # (the path PR #190860 fixed). + del meta + del tensor + gc.collect() + torch.cuda.empty_cache() + torch.cuda.synchronize() + _log("freed segment (destructor exercised)") + + +def _dist_worker(rank: int, world_size: int) -> None: + import gc + + import torch + import torch.distributed as dist + from torch.multiprocessing.reductions import reduce_tensor + + os.environ.setdefault("MASTER_ADDR", MASTER_ADDR) + os.environ.setdefault("MASTER_PORT", MASTER_PORT) + + torch.cuda.set_device(rank) + dist.init_process_group(backend="gloo", rank=rank, world_size=world_size) + try: + device = torch.device("cuda", rank) + if rank == 0: + torch.cuda.memory._set_allocator_settings("expandable_segments:True") + torch.cuda.empty_cache() + mib = TENSOR_ELEMENTS * 4 / 1024 / 1024 + _log(f"rank0 producer: allocating {mib:.0f} MiB expandable segment on {device}") + tensor = torch.randn(TENSOR_ELEMENTS, device=device) + meta = reduce_tensor(tensor) + _log("rank0 producer: exported handle, broadcasting to consumer") + dist.broadcast_object_list([meta], src=0) + dist.barrier() + del meta + del tensor + gc.collect() + torch.cuda.empty_cache() + _log("rank0 producer: freed segment (destructor exercised)") + else: + # Do NOT pre-allocate: a consumer that has never touched the + # allocator is the #179220 regression shape. + _log(f"rank{rank} consumer: receiving handle (no pre-allocation)") + recv: list[object] = [None] + dist.broadcast_object_list(recv, src=0) + func, args = recv[0] # type: ignore[misc] + args = list(args) + # args[6] is storage_device (rebuild_cuda_tensor): import onto this + # consumer's own GPU, exercising the cross-GPU fabric import. + args[6] = rank + tensor = func(*args) + _log(f"rank{rank} consumer: imported handle onto {device}") + dist.barrier() + del tensor + gc.collect() + torch.cuda.empty_cache() + _log(f"rank{rank} consumer: freed imported segment") + torch.cuda.synchronize() + finally: + dist.destroy_process_group() + + +def _multiprocess_workload() -> None: + import torch + import torch.multiprocessing as mp + + if not torch.cuda.is_available(): + raise SystemExit("CUDA is not available; cannot run the fabric probe") + device_count = torch.cuda.device_count() + if device_count < 2: + raise SystemExit(f"multiprocess mode needs >=2 GPUs, found {device_count}; use --single-process") + + world_size = 2 + print(f"{MODE_MARKER} multiprocess ({world_size} GPUs)", flush=True) + os.environ.setdefault("MASTER_ADDR", MASTER_ADDR) + os.environ.setdefault("MASTER_PORT", MASTER_PORT) + mp.spawn(_dist_worker, args=(world_size,), nprocs=world_size, join=True) + + +def _run_workload(single_process: bool) -> int: + import torch + + if single_process or torch.cuda.device_count() < 2: + _single_process_workload() + else: + _multiprocess_workload() + return 0 + + +def _verdict(log: str, workload_rc: int) -> tuple[bool, str]: + fabric_share = LOG_FABRIC_SHARE in log + posix_share = LOG_POSIX_SHARE in log + fabric_import = LOG_FABRIC_IMPORT in log + posix_import = LOG_POSIX_IMPORT in log + multiprocess = f"{MODE_MARKER} multiprocess" in log + + if workload_rc != 0: + return False, f"workload process exited non-zero ({workload_rc}); see logs above" + if not fabric_share and not posix_share: + return False, ( + "SHARE path never exercised: no fabric/posix share log seen. The torch build may " + "lack expandable-segments IPC, the segment was not created, or C++ INFO logging was " + "not enabled (TORCH_CPP_LOG_LEVEL=INFO). Result is inconclusive, not a pass." + ) + if posix_share: + return False, "fell back to POSIX fd on the SHARE (export) path - fabric was NOT used" + if multiprocess and posix_import: + return False, "fell back to POSIX fd on the IMPORT path - fabric was NOT used" + if multiprocess and not fabric_import: + return False, "consumer never imported via fabric; cross-GPU round trip did not complete" + return True, "fabric handle used on the expandable-segments IPC path" + + +def _evaluate(log: str, workload_rc: int) -> int: + signals = { + "fabric share": LOG_FABRIC_SHARE in log, + "posix share (fallback)": LOG_POSIX_SHARE in log, + "fabric import": LOG_FABRIC_IMPORT in log, + "posix import (fallback)": LOG_POSIX_IMPORT in log, + "isFabricSupported ok": LOG_FABRIC_OK in log, + "isFabricSupported fell back": LOG_FABRIC_FALLBACK in log, + } + _log("captured C++ log signals:") + for name, seen in signals.items(): + print(f" {'FOUND' if seen else '- '} {name}", flush=True) + + passed, reason = _verdict(log, workload_rc) + if passed: + print(f"{VERDICT_MARKER} PASS - {reason}", flush=True) + return 0 + print(f"{VERDICT_MARKER} FAIL - {reason}", flush=True) + return 1 + + +def _run_driver(single_process: bool) -> int: + cmd = [sys.executable, os.path.abspath(__file__), "--run"] + if single_process: + cmd.append("--single-process") + _log("launching workload: " + " ".join(cmd)) + + proc = subprocess.Popen( + cmd, + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + text=True, + bufsize=1, + ) + captured: list[str] = [] + if proc.stdout is not None: + for line in proc.stdout: + sys.stdout.write(line) + sys.stdout.flush() + captured.append(line) + workload_rc = proc.wait() + return _evaluate("".join(captured), workload_rc) + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument( + "--run", + action="store_true", + help="internal: execute the torch workload (invoked by the driver)", + ) + parser.add_argument( + "--single-process", + action="store_true", + help="force the single-process, single-GPU workload", + ) + args = parser.parse_args() + + if args.run: + return _run_workload(single_process=args.single_process) + return _run_driver(single_process=args.single_process) + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/osdc/modules/arc-runners/scripts/python/generate_runners.py b/osdc/modules/arc-runners/scripts/python/generate_runners.py index c51e187f..cf48d720 100755 --- a/osdc/modules/arc-runners/scripts/python/generate_runners.py +++ b/osdc/modules/arc-runners/scripts/python/generate_runners.py @@ -251,6 +251,7 @@ def generate_runner( max_burst_capacity = runner.get("max_burst_capacity", 0) hud_failure_base_capacity = runner.get("hud_failure_base_capacity", 0) node_fleet_override = runner.get("node_fleet") + imex_channels = runner.get("imex_channels") proactive_cap = cluster_config.get("proactive_capacity_max") if proactive_cap is not None: proactive_capacity = min(proactive_capacity, proactive_cap) @@ -377,6 +378,14 @@ def generate_runner( gpu_request = "" gpu_limit = "" + # Optional NVIDIA_IMEX_CHANNELS env on the workflow job container. Trails the + # preceding env value line inline so an unset field leaves that line — and the + # whole rendered runner — byte-identical to output without this feature. + if imex_channels is not None: + imex_env = f'\n - name: NVIDIA_IMEX_CHANNELS\n value: "{imex_channels}"' + else: + imex_env = "" + # Runner class isolation snippets (workflow pod only) # Release runners: required affinity to target release nodes # Regular runners: required affinity to avoid release nodes (DoesNotExist) @@ -442,6 +451,7 @@ def generate_runner( "{{GPU_NODE_SELECTOR_AFFINITY}}": gpu_node_selector_affinity, "{{GPU_REQUEST}}": gpu_request, "{{GPU_LIMIT}}": gpu_limit, + "{{IMEX_ENV}}": imex_env, "{{MODULE_NAME}}": module_name, "{{RUNNER_IMAGE}}": runner_image, # HF cache write target (used by the ci-refresh-hf-cache workflow's OIDC diff --git a/osdc/modules/arc-runners/scripts/python/test_generate_runners.py b/osdc/modules/arc-runners/scripts/python/test_generate_runners.py index d1ede96a..ae476e7f 100644 --- a/osdc/modules/arc-runners/scripts/python/test_generate_runners.py +++ b/osdc/modules/arc-runners/scripts/python/test_generate_runners.py @@ -200,6 +200,7 @@ def make_def_file( node_fleet=None, scheduler_name=None, fresh_multiplier=None, + imex_channels=None, ): """Write a runner def YAML and return the path. @@ -232,6 +233,8 @@ def make_def_file( runner["scheduler_name"] = scheduler_name if fresh_multiplier is not None: runner["fresh_multiplier"] = fresh_multiplier + if imex_channels is not None: + runner["imex_channels"] = imex_channels content = {"runner": runner} p = tmp_path / f"{name}.yaml" p.write_text(yaml.dump(content, default_flow_style=False)) @@ -2917,6 +2920,49 @@ def test_main_keeps_when_cluster_has_pypi_cache_module(self, tmp_path, monkeypat assert "# END_PYPI_CACHE" not in content +class TestImexChannels: + """Optional NVIDIA_IMEX_CHANNELS env var on the workflow job container.""" + + def _cluster_config(self): + return { + "github_config_url": "url", + "github_secret_name": "secret", + "runner_name_prefix": "", + } + + def test_imex_channels_set_injects_env(self, tmp_path, real_template): + """imex_channels: 0 renders NVIDIA_IMEX_CHANNELS="0" on the job container. + + 0 is deliberate: it is falsy, so this also proves the gate uses + ``is not None`` rather than truthiness (a truthiness check would drop 0). + """ + def_file = make_def_file( + tmp_path, "imex-runner", "p6-b200.48xlarge", 22, 225, gpu=1, disk_size=600, imex_channels=0 + ) + output_dir = tmp_path / "out" + output_dir.mkdir() + + assert generate_runner(def_file, real_template, self._cluster_config(), output_dir, "arc-runners") is True + + docs = list(yaml.safe_load_all((output_dir / "imex-runner.yaml").read_text())) + cm_data = yaml.safe_load(docs[1]["data"]["job-pod.yaml"]) + env_vars = {e["name"]: e["value"] for e in cm_data["spec"]["containers"][0]["env"]} + assert env_vars["NVIDIA_IMEX_CHANNELS"] == "0" + + def test_imex_channels_unset_omits_env(self, tmp_path, real_template): + """Without imex_channels, NVIDIA_IMEX_CHANNELS is absent from the job container.""" + def_file = make_def_file(tmp_path, "plain-runner", "c7i.24xlarge", 4, 16) + output_dir = tmp_path / "out" + output_dir.mkdir() + + assert generate_runner(def_file, real_template, self._cluster_config(), output_dir, "arc-runners") is True + + docs = list(yaml.safe_load_all((output_dir / "plain-runner.yaml").read_text())) + cm_data = yaml.safe_load(docs[1]["data"]["job-pod.yaml"]) + env_names = [e["name"] for e in cm_data["spec"]["containers"][0]["env"]] + assert "NVIDIA_IMEX_CHANNELS" not in env_names + + def _fresh_multiplier_env_value(output_dir, runner_name): docs = list(yaml.safe_load_all((output_dir / f"{runner_name}.yaml").read_text())) helm = docs[0] diff --git a/osdc/modules/arc-runners/templates/runner.yaml.tpl b/osdc/modules/arc-runners/templates/runner.yaml.tpl index aadd18ff..824d3b38 100644 --- a/osdc/modules/arc-runners/templates/runner.yaml.tpl +++ b/osdc/modules/arc-runners/templates/runner.yaml.tpl @@ -461,7 +461,7 @@ data: value: "{{HF_CACHE_REGION}}" # END_HF_CACHE - name: TORCH_CI_MAX_MEMORY - value: "{{MEMORY_BYTES}}" + value: "{{MEMORY_BYTES}}"{{IMEX_ENV}} # Workflow container gets the actual compute resources resources: requests: diff --git a/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh b/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh index 56380cae..a117454f 100755 --- a/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh +++ b/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh @@ -119,8 +119,8 @@ systemctl enable cpu-performance.service # Enable GPU persistence mode for consistent performance nvidia-smi -pm 1 || true -# ---- Fabric Manager / IMEX: do nothing here ---- -# Background for future maintainers: +# ---- Fabric Manager / IMEX ---- +# Fabric Manager daemon: deliberately left untouched here. # * The AMI (amazon-eks-node-al2023-x86_64-nvidia-*) already installs and # enables nvidia-fabricmanager and nvidia-persistenced. systemd brings FM # up later in boot, after the GPU-resident "local FM" instances have @@ -131,15 +131,81 @@ nvidia-smi -pm 1 || true # NVLink Inband. During cloud-init they are usually not ready yet, so FM # exits with "config error type 8 / not all local fabric manager # instances finished their configuration", taking the whole node out. -# * IMEX support on this AMI is compiled into nvidia.ko (char device class -# nvidia-caps-imex-channels, major 242). There is no separate -# nvidia-caps-imex-channels.ko; `modprobe nvidia-caps-imex-channels` -# will always fail. -# * Single-node NVLS multicast on p5 does not require an IMEX channel -# device - NCCL uses POSIX FD handles intranode. IMEX channels are only -# needed for multi-node NVLink (MNNVL / GB200 UltraServers). -# If a fresh provision ever shows FM failing on the normal boot path (not -# cloud-init), prefer a systemd drop-in with After=nvidia-persistenced.service -# and Restart=on-failure over taking control of FM here. +# If a fresh provision ever shows FM failing on the normal boot path (not +# cloud-init), prefer a systemd drop-in with After=nvidia-persistenced.service +# and Restart=on-failure over taking control of FM here. +# +# IMEX channel device: created below, for the current boot and (via an enabled +# oneshot) across reboots. +# * IMEX is compiled into nvidia.ko (char device class +# nvidia-caps-imex-channels; its char major is assigned dynamically, so it +# is read from /proc/devices rather than hardcoded). There is no separate +# nvidia-caps-imex-channels.ko - `modprobe nvidia-caps-imex-channels` always +# fails. +# * The driver does not auto-create the channel node, yet a single-node CUDA +# VMM fabric handle (CU_MEM_HANDLE_TYPE_FABRIC) - used by PyTorch +# expandable-segments IPC - needs /dev/nvidia-caps-imex-channels/channel0 to +# exist even on one host. NCCL's intra-node path uses POSIX FD handles and +# does not need a channel; the CUDA VMM fabric-handle path does, even on a +# single host. Fabric Manager is not required for this - the channel node +# plus the driver's single-node fabric are sufficient. +# +# The creation logic is written to a helper so the current boot and the oneshot +# share one implementation. It cannot be an inline ExecStart=/bin/bash -c with +# awk: systemd expands every unescaped `$` in ExecStart itself (undefined -> +# empty), so awk field refs and `$(...)` would be mangled. A plain script path +# has nothing for systemd to expand. +IMEX_HELPER=/usr/local/sbin/nvidia-imex-channel.sh +cat >"$IMEX_HELPER" <<'EOFHELPER' +#!/bin/bash +CHANNEL=/dev/nvidia-caps-imex-channels/channel0 +if [ -e "$CHANNEL" ]; then + echo "[h100-node-setup] IMEX $CHANNEL already present" + exit 0 +fi +MAJOR=$(awk '$2=="nvidia-caps-imex-channels"{print $1}' /proc/devices) +if [ -z "$MAJOR" ]; then + echo "ERROR: [h100-node-setup] char class nvidia-caps-imex-channels absent from /proc/devices; cannot create $CHANNEL" >&2 + exit 1 +fi +if mkdir -p /dev/nvidia-caps-imex-channels \ + && mknod "$CHANNEL" c "$MAJOR" 0 \ + && chmod 0666 "$CHANNEL"; then + echo "[h100-node-setup] created IMEX $CHANNEL (char major $MAJOR)" +else + echo "ERROR: [h100-node-setup] failed to create IMEX $CHANNEL (char major $MAJOR)" >&2 + exit 1 +fi +EOFHELPER +chmod 0755 "$IMEX_HELPER" + +# Create the channel for the current boot now. Run under `if` so a creation +# failure logs but never trips `set -e`: a node without fabric handles is +# recoverable, a failed cloud-init is not. +if ! "$IMEX_HELPER"; then + echo "ERROR: [h100-node-setup] IMEX channel not created for the current boot (see above); continuing" >&2 +fi + +cat >/etc/systemd/system/nvidia-imex-channel.service < Date: Fri, 24 Jul 2026 15:54:29 -0700 Subject: [PATCH 2/4] Document H100 fabric handles and IMEX channels MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add docs/h100-fabric-handles-imex-channels.md covering CUDA CU_MEM_HANDLE_TYPE_FABRIC support on the OSDC H100 fleet - Record live-node verification (p5.48xlarge, 2026-07-24): fabric allocate/export/import round-trip passes; only the IMEX channel device is missing, a per-node config gap, not a hardware gap - Detail the fabric-enablement design: node-side mknod channel creation + unprivileged pod injection via NVIDIA_IMEX_CHANNELS, gated per-def to H100, with fails-closed ordering - Correct the Fabric Manager / IMEX race notes and the "multi-node only" script comment for the CUDA fabric-handle path - Link the new doc from the CLAUDE.md docs index Notes: Prompted by pytorch/pytorch#190860 (fabric-handle destructor crash). Debunks the claim that AWS p5 reports fabric NOT_SUPPORTED and that fabric CI needs GB200 hardware or the nvidia-imex daemon. Documentation only — no infrastructure or runner behavior changes in this commit. Signed-off-by: Jean Schmidt --- osdc/CLAUDE.md | 1 + .../docs/h100-fabric-handles-imex-channels.md | 228 ++++++++++++++++++ 2 files changed, 229 insertions(+) create mode 100644 osdc/docs/h100-fabric-handles-imex-channels.md diff --git a/osdc/CLAUDE.md b/osdc/CLAUDE.md index 89a19484..b1cdd7b1 100644 --- a/osdc/CLAUDE.md +++ b/osdc/CLAUDE.md @@ -79,3 +79,4 @@ Reference documentation in `docs/`: | `docs/node-warmup-and-scheduling-gates.md` | Full node initialization sequence — taints, DaemonSets, init containers before job scheduling | | `docs/arc-fork-build-deploy.md` | ARC fork (jeanschmidt/actions-runner-controller) build/release workflow and chart publishing | | `docs/pypi-package-cache.md` | PyPI wheel cache architecture, slug naming, S3 layout, runner integration | +| `docs/h100-fabric-handles-imex-channels.md` | Enabling CUDA fabric handles (`CU_MEM_HANDLE_TYPE_FABRIC`) on H100 runners — IMEX channel creation, the gated fabric runner, and verification | diff --git a/osdc/docs/h100-fabric-handles-imex-channels.md b/osdc/docs/h100-fabric-handles-imex-channels.md new file mode 100644 index 00000000..f3836d31 --- /dev/null +++ b/osdc/docs/h100-fabric-handles-imex-channels.md @@ -0,0 +1,228 @@ +# H100 Fabric Handles and IMEX Channels (OSDC) + +## Context + +PyTorch [pytorch/pytorch#190860](https://github.com/pytorch/pytorch/pull/190860) fixed a +destructor crash in `ExpandableSegment::unmapHandles` — `std::bad_variant_access` when the +cached `shareable_handle` holds a `CUmemFabricHandle` instead of a POSIX fd. A follow-up +question asked whether OSDC could run CI that exercises the CUDA **fabric handle** path, and +whether that requires new (multi-node NVLink / GB200) hardware. An automated bot comment on +the PR asserted that standard AWS `p5` (H100) instances report GPU fabric state +`NOT_SUPPORTED`, and that fabric CI would need fabric-provisioned hardware plus the +`nvidia-imex` daemon. + +This doc records what is actually true on the OSDC H100 fleet, verified against a live node +on 2026-07-24. + +## TL;DR + +- **OSDC H100 (`p5.48xlarge`) nodes are fabric-capable today.** `CU_MEM_HANDLE_TYPE_FABRIC` + allocate → export → import → free was verified working on a live node. **No new hardware, + no GB200 NVL72, no `nvidia-imex` daemon.** +- **The full production injection path is verified end-to-end** — a host-created channel plus + `NVIDIA_IMEX_CHANNELS` on an *unprivileged* pod: the pod starts and the fabric probe passes. +- The bot's "p5 reports `NOT_SUPPORTED`" claim is **false** — a live p5 reports fabric + `State: Completed / Status: Success`. +- The **only** missing piece is the **IMEX channel device** + (`/dev/nvidia-caps-imex-channels/channel0`), which is absent on our nodes by design. It is + a per-node/per-container config gap, not a hardware gap. + +## Background: fd vs fabric handles + +When a CUDA process shares a VMM allocation with another process, the exported handle is one +of two kinds: +- **POSIX fd** (`CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR`) — works within a single machine. + This is what every OSDC runner uses today. +- **Fabric** (`CU_MEM_HANDLE_TYPE_FABRIC`, a `CUmemFabricHandle`, CUDA 12.4+) — rides the + NVLink fabric. PyTorch's expandable-segments IPC / SymmetricMemory selects this when + `get_fabric_access()` is true. + +The #190860 crash only occurs on the fabric branch: `unmapHandles` did +`close(std::get(*h.shareable_handle))`, which throws from a destructor when the variant +holds a fabric handle (fixed by switching to the non-throwing `std::get_if`). + +## Finding: verified on a live node (2026-07-24) + +Node `ip-10-8-105-26` (`p5.48xlarge`, 8× H100 80GB) on cluster `meta-prod-aws-uw1`, driver +`580.159.03`, CUDA `13.0`: + +- `nvidia-smi -q` GPU Fabric: `State: Completed`, `Status: Success`, `CliqueId: 0`, all-zero + `ClusterUUID` (a standalone single-node fabric island — not part of a multi-node clique). +- With an IMEX channel device present (`mknod .../channel0 c 242 0`, mode `0666`), a minimal + CUDA driver-API program ran the full cycle successfully: + `cuMemCreate(requestedHandleTypes=CU_MEM_HANDLE_TYPE_FABRIC)` → + `cuMemExportToShareableHandle(FABRIC)` → `cuMemImportFromShareableHandle(FABRIC)` → + `cuMemRelease` → exit 0 (`FABRIC_PROBE: PASS`). +- `CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_FABRIC_SUPPORTED` returned 1 (informational only — + documented-unreliable; the export/import round-trip is the authoritative test). + +### Requirements for single-node fabric handles on H100 +1. **NVSwitch node** — `p5` (H100) and `p6-b200` (B200) qualify; each is an 8-GPU intra-node + NVSwitch island. +2. **Fabric Manager running** — `nv-fabricmanager`; already running via the EKS NVIDIA AMI, + which is why fabric `State` is `Completed`. FM configures NVSwitch/fabric; it does **not** + create IMEX channels. +3. **IMEX channel device** — `/dev/nvidia-caps-imex-channels/channelN` (char major 242 on + this AMI), created via `mknod` or the `NVreg_CreateImexChannel0=1` driver param, and + accessible (`0666`) to the process. Required even single-node: the driver enforces channel + access on `cuMemImportFromShareableHandle(FABRIC)` (else `CUDA_ERROR_NOT_PERMITTED`). +4. **NOT required single-node:** the `nvidia-imex` daemon (it only coordinates cross-node / + multi-node NVLink domains). + +### Current state on OSDC H100 nodes +- Fabric Manager: running. Fabric state: `Completed`. +- IMEX channel device: **absent** (not created by default; the node bootstrap deliberately + does not create it). +- Net effect: `c10::cuda::get_fabric_access()` (PyTorch `c10/cuda/PeerToPeerAccess.cpp:164`) + returns **false** — its `isFabricSupported()` probe's `cuMemImportFromShareableHandle(FABRIC)` + fails with `CUDA_ERROR_NOT_PERMITTED` when no channel is accessible — so the allocator uses + the POSIX-fd path. That is why no OSDC CI has ever hit the #190860 fabric-destructor bug. + +## The documented Fabric Manager / IMEX race + +`modules/nodepools-h100/scripts/h100-node-setup.sh:122-143` (identical block in +`modules/nodepools-b200/scripts/b200-node-setup.sh`), added by osdc#552 (2026-05-12) after +osdc#541's eager approach caused a 100% H100 node-kill rate: + +- **The race:** starting `nvidia-fabricmanager` from cloud-init races the GPUs' on-die + "local FM" init. Host FM finishes NVSwitch routing in seconds, then waits up to the 30 s + GFM Wait Timeout for the GPU local-FM instances to register over NVLink Inband. From + cloud-init they are not ready yet → FM aborts with "config error type 8 / not all local + fabric manager instances finished their configuration" → the hard-failing setup script + fails → the node never joins → Karpenter reaps it. +- **The fix:** do nothing in cloud-init. The AMI already `systemctl enable`d FM, so systemd + starts it later in normal boot (GPUs ready), completing in <1 s. +- **Scope:** the race is specific to **starting the FM daemon early**. Creating an IMEX + channel device does not start FM and does not re-trigger this race. + +**Correction to the script comment.** The comment states "IMEX channels are only needed for +multi-node NVLink." That is true for **NCCL** (intra-node NCCL/NVLS multicast uses POSIX-fd +handles) but **not** for CUDA `CU_MEM_HANDLE_TYPE_FABRIC` (PyTorch expandable-segments IPC / +SymmetricMemory), which needs the channel even on a single node. The comment also correctly +notes that `modprobe nvidia-caps-imex-channels` always fails — IMEX is compiled into +`nvidia.ko` (there is no separate `.ko`); the channel is created via `mknod` or the +`NVreg_CreateImexChannel0` module param, not by loading a module. + +## Enabling fabric handles on the existing fleet (design) + +No new hardware required. Two pieces: + +### 1. Create the channel on the node +- **Runtime `mknod`** (fits ephemeral Karpenter nodes): + `mknod /dev/nvidia-caps-imex-channels/channel0 c 0 && chmod 0666`, deriving + `` from `/proc/devices` (242 on the current AMI). Add to `h100-node-setup.sh` after + the `nvidia-smi -pm 1` line (:120). The script runs after the driver is loaded, so a + `modprobe.d` drop-in would not take effect until reboot — hence runtime `mknod`, not + modprobe, at cloud-init time. +- **Reboot-persistent alternative:** `options nvidia NVreg_CreateImexChannel0=1` in + `/etc/modprobe.d/` + initramfs rebuild (auto-creates a world-accessible channel0 on every + driver load). Heavier; for a baked AMI or fleet-wide rollout. +- Neither starts Fabric Manager, so neither re-triggers the documented race. + +### 2. Inject the channel into runner job pods +- OSDC uses the **stock** `k8s-device-plugin` (`base/kubernetes/nvidia-device-plugin.yaml:45-48`, + envvar strategy) with the NVIDIA container-toolkit runtime hook. Node toolkit config + (`/etc/nvidia-container-runtime/config.toml`) has + `accept-nvidia-visible-devices-envvar-when-unprivileged = true` (default) and CDI enabled, + so **unprivileged** pods can request IMEX channels via env. +- Set `NVIDIA_IMEX_CHANNELS="0"` on the job container (the toolkit maps it to CDI + `runtime.nvidia.com/imex-channel=0`). Equivalent alternatives: CDI device + `nvidia.com/imex-channel=0`, or the `/dev/null → /var/run/nvidia-container-devices/imex/channel0` + volume-mount. +- **Wire-up point:** the job-container env block in + `modules/arc-runners/templates/runner.yaml.tpl:431-464`, driven by a new per-def field + (e.g. `imex_channels`) read in `modules/arc-runners/scripts/python/generate_runners.py` + (~:242, replacements dict ~:429-467) → a new `{{IMEX_ENV}}` placeholder. Set the field only + on `modules/arc-runners-h100/defs/*.yaml` to gate to H100. + +### FAILS CLOSED — ordering matters +If `NVIDIA_IMEX_CHANNELS` is set on a pod but the host channel does not exist, the toolkit +stat's the host device, fails, and the pod goes to **`StartError`**: +``` +failed to apply edits for device "runtime.nvidia.com/imex-channel=0": +failed to stat CDI host device "/dev/nvidia-caps-imex-channels/channel0": no such file or directory +``` +So channel creation on the node MUST land before/with the env injection, or **every** H100 +runner pod fails to start. Conversely, creating the host channel is harmless to pods that do +NOT request it: injection is per-container, so their `get_fabric_access()` stays false and +their behavior is unchanged. This is what makes per-def gating safe. + +## How this was verified (2026-07-24) + +- A **self-contained privileged pod requesting all 8 GPUs** so it owns a whole node (no CI + co-location). A 1-GPU request instead co-locates onto a busy CI node — avoid. An 8-GPU + request routes to the `p5-large` Karpenter pool and scales a fresh `reserved` node. +- Image `nvcr.io/nvidia/cuda:13.0.0-devel-ubuntu24.04` (pulls via the node's Harbor + `nvcr.io` mirror; `-devel` ships `nvcc`). +- The pod created the channel in-container (`mknod`), compiled a driver-API probe + (`nvcc ... -L/usr/local/cuda/lib64/stubs -lcuda`), and ran it → `FABRIC_PROBE: PASS`. +- Isolation / scheduling facts: the `p5-48xlarge` pool is `consolidationPolicy: WhenEmpty`, + `consolidateAfter: 1h`; fresh p5 nodes carry a `git-cache-not-ready:NoSchedule` startup + taint (must be tolerated); a pod holding a non-DaemonSet workload pins its node against + consolidation. + +### Production injection path — verified (2026-07-24) +The unprivileged toolkit-injection path was confirmed end-to-end on an isolated `p5-large` +node (`ip-10-8-94-202`) with a single pod: a **privileged init container** `mknod`'d the host +channel (`crw-rw-rw- 242,0 channel0`), then the **unprivileged main container** +(`NVIDIA_IMEX_CHANNELS=0`, no privileged securityContext) **started** — no `StartError`, in +contrast to the bare attempt with no host channel — with +`/dev/nvidia-caps-imex-channels/channel0` injected by the toolkit, and the fabric probe +returned `FABRIC_PROBE: PASS`. This is the exact production wiring: node bootstrap creates the +host channel, the toolkit injects it into the unprivileged runner pod via the env, and the +unprivileged process uses fabric handles successfully. Caveat: the test container ran as +uid 0 (root) but *unprivileged* (no privileged securityContext); the channel's `0666` mode +makes it accessible to non-root runner UIDs (e.g. uid 1000) as well. + +## Hardware scope +- **H100 (`p5.48xlarge`): verified.** +- **B200 (`p6-b200.48xlarge`): expected identical** (same NVSwitch single-node architecture, + same AMI and node bootstrap) but not yet tested. +- **GB200 NVL72 (cross-node MNNVL):** a different regime that *does* need the `nvidia-imex` + daemon; not present in OSDC and not required for single-node fabric handles. On AWS this is + P6e-GB200 UltraServers (Capacity Blocks only) — overkill for exercising this code path. + +## Key file pointers + +OSDC (`ci-infra/osdc`): +- `modules/nodepools-h100/scripts/h100-node-setup.sh:122-143` — FM/IMEX "do nothing" + rationale; `nvidia-smi -pm 1` at :120. +- `modules/arc-runners/templates/runner.yaml.tpl:431-464` — job-container env block + (injection point). +- `modules/arc-runners/scripts/python/generate_runners.py` — ~:242 (read def field), + :359-378 (GPU tolerations/affinity/requests logic), ~:429-467 (replacements dict). +- `base/kubernetes/nvidia-device-plugin.yaml:45-48` — stock device plugin, no IMEX config. +- `modules/nodepools-h100/defs/{p5,p5-large}.yaml`, `modules/arc-runners-h100/defs/*.yaml`. + +PyTorch (`pytorch/pytorch`): +- `c10/cuda/PeerToPeerAccess.cpp:164` (`get_fabric_access`), :110 (`isFabricSupported`). +- `c10/cuda/CUDACachingAllocator.cpp:826` (`detectHandleType`), :620-638 (`share`), :942 + (destructor `std::get` fixed by #190860), :993 (`shareable_handle` variant). +- `test/distributed/test_p2p_ipc.py::test_p2p_ipc_expandable_segments` (currently + `@unittest.skip`, #189879). +- PR: https://github.com/pytorch/pytorch/pull/190860 + +## Sources (NVIDIA) +- CUDA VMM — fabric handles single- and multi-node: + https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/virtual-memory-management.html +- CUDA Driver API — `CUDA_ERROR_NOT_PERMITTED` on fabric import: + https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__VA.html +- IMEX channels guide: https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/imexchannels.html +- IMEX overview (daemon = multi-node only): + https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/overview.html +- Fabric Manager user guide (single-node HGX/DGX scope): + https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html +- NVML fabric-state defs: https://docs.nvidia.com/deploy/nvml-api/group__nvmlFabricDefs.html +- Container Toolkit CDI / IMEX: + https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html +- Single-node fabric-handle confirmation (NVIDIA staff, dev forum): + https://forums.developer.nvidia.com/t/cudevicegetattribute-shows-i-can-use-fabric-handle-but-actually-i-cannot/336426 + +## Source / provenance +Verified on `meta-prod-aws-uw1` (`p5.48xlarge`, driver 580.159.03, CUDA 13.0) on 2026-07-24 +via two isolated probe pods: a self-contained privileged pod that created the channel +in-container and ran the fabric probe (node `ip-10-8-105-26`), and a two-container pod that +created the channel on the host and ran the probe from an unprivileged container with the +channel toolkit-injected via `NVIDIA_IMEX_CHANNELS` (node `ip-10-8-94-202`) — both +`FABRIC_PROBE: PASS`. Plus source review of OSDC node/runner modules and PyTorch `c10/cuda`. +Prompted by pytorch/pytorch#190860. From 7353b148f6dbc21ea01613c4dc1d570b3cf038cd Mon Sep 17 00:00:00 2001 From: Jean Schmidt Date: Fri, 24 Jul 2026 16:11:23 -0700 Subject: [PATCH 3/4] Expand H100 fabric runners into GPU-size family - Add 1/2/4/8-GPU fabric runner defs (22-225, 44-450-2, 88-900-4, 176-1800-8) - Replace the single dedicated 2-GPU fabric runner (max_runners: 1) with the sized family - Give the 8-GPU full-node runner its own p5-large fleet to avoid single-numa-node TopologyAffinityError - Set per-def max_runners with a meta-prod-aws-uw1 override sized to the reserved 8-GPU node capacity - Remove the manual fabric-IPC validation probe (README, pod manifest, fabric_probe.py) Notes: The prior fabric rollout shipped one low-cap 2-GPU runner plus a manual probe to guard against silent POSIX-fd fallback. This generalizes fabric access to every GPU shape so jobs can request 1, 2, 4, or 8 H100s, with caps derived from the fixed reservation (8 GPUs/node): 8/4/2/1 concurrent runners by GPU count, scaled up on meta-prod-aws-uw1. The 8-GPU pod cannot satisfy the shared p5-48xlarge pool's single-numa-node policy across two NUMA nodes, so it runs on a dedicated p5-large fleet. The validation probe was a one-off manual check and is dropped now that fabric is standard across the fleet. Signed-off-by: Jean Schmidt --- .../defs/l-bx86iamx-176-1800-h100-fab-8.yaml | 19 ++ .../defs/l-x86iamx-22-225-h100-fab.yaml | 15 + .../defs/l-x86iamx-44-450-h100-fab-2.yaml | 14 + .../defs/l-x86iamx-44-450-h100-fab.yaml | 12 - .../defs/l-x86iamx-88-900-h100-fab-4.yaml | 14 + .../arc-runners-h100/validation/README.md | 137 -------- .../validation/fabric-validation-pod.yaml | 101 ------ .../validation/fabric_probe.py | 293 ------------------ 8 files changed, 62 insertions(+), 543 deletions(-) create mode 100644 osdc/modules/arc-runners-h100/defs/l-bx86iamx-176-1800-h100-fab-8.yaml create mode 100644 osdc/modules/arc-runners-h100/defs/l-x86iamx-22-225-h100-fab.yaml create mode 100644 osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab-2.yaml delete mode 100644 osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab.yaml create mode 100644 osdc/modules/arc-runners-h100/defs/l-x86iamx-88-900-h100-fab-4.yaml delete mode 100644 osdc/modules/arc-runners-h100/validation/README.md delete mode 100644 osdc/modules/arc-runners-h100/validation/fabric-validation-pod.yaml delete mode 100644 osdc/modules/arc-runners-h100/validation/fabric_probe.py diff --git a/osdc/modules/arc-runners-h100/defs/l-bx86iamx-176-1800-h100-fab-8.yaml b/osdc/modules/arc-runners-h100/defs/l-bx86iamx-176-1800-h100-fab-8.yaml new file mode 100644 index 00000000..e9bbb539 --- /dev/null +++ b/osdc/modules/arc-runners-h100/defs/l-bx86iamx-176-1800-h100-fab-8.yaml @@ -0,0 +1,19 @@ +# ARC runner definition: l-bx86iamx-176-1800-h100-fab-8 (8x NVIDIA H100, full node) +# Scheduled on NodePool fleet: p5-large (from nodepools-h100 module). +# Uses its OWN fleet — NOT the shared p5-48xlarge pool — because that pool pins +# topologyManagerPolicy: single-numa-node, which an all-8-GPU pod can never +# satisfy on a 2-NUMA-node p5.48xlarge (TopologyAffinityError). See +# nodepools-h100/defs/p5-large.yaml for the full rationale. +runner: + name: l-bx86iamx-176-1800-h100-fab-8 + instance_type: p5.48xlarge + node_fleet: p5-large + disk_size: 200 + vcpu: 176 + memory: 1800Gi + gpu: 8 + imex_channels: 0 + # Fixed-capacity cap: 1 reserved 8-GPU node / 8 GPUs per runner = 1 concurrent runner. + max_runners: + default: 1 + meta-prod-aws-uw1: 6 diff --git a/osdc/modules/arc-runners-h100/defs/l-x86iamx-22-225-h100-fab.yaml b/osdc/modules/arc-runners-h100/defs/l-x86iamx-22-225-h100-fab.yaml new file mode 100644 index 00000000..f9fd8b0e --- /dev/null +++ b/osdc/modules/arc-runners-h100/defs/l-x86iamx-22-225-h100-fab.yaml @@ -0,0 +1,15 @@ +# ARC runner definition: l-x86iamx-22-225-h100-fab (1x NVIDIA H100) +# Scheduled on NodePool: p5-48xlarge (from nodepools-h100 module) +# Resource math: 176 usable vCPU / 8 GPUs = 22 vCPU/GPU, 1800 GiB / 8 = 225 GiB/GPU +runner: + name: l-x86iamx-22-225-h100-fab + instance_type: p5.48xlarge + disk_size: 200 + vcpu: 22 + memory: 225Gi + gpu: 1 + imex_channels: 0 + # Fixed-capacity cap: 1 reserved 8-GPU node / 1 GPU per runner = 8 concurrent runners. + max_runners: + default: 8 + meta-prod-aws-uw1: 48 diff --git a/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab-2.yaml b/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab-2.yaml new file mode 100644 index 00000000..9b762183 --- /dev/null +++ b/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab-2.yaml @@ -0,0 +1,14 @@ +# ARC runner definition: l-x86iamx-44-450-h100-fab-2 (2x NVIDIA H100) +# Scheduled on NodePool: p5-48xlarge (from nodepools-h100 module) +runner: + name: l-x86iamx-44-450-h100-fab-2 + instance_type: p5.48xlarge + disk_size: 200 + vcpu: 44 + memory: 450Gi + gpu: 2 + imex_channels: 0 + # Fixed-capacity cap: 1 reserved 8-GPU node / 2 GPUs per runner = 4 concurrent runners. + max_runners: + default: 4 + meta-prod-aws-uw1: 24 diff --git a/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab.yaml b/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab.yaml deleted file mode 100644 index ec50e672..00000000 --- a/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab.yaml +++ /dev/null @@ -1,12 +0,0 @@ -# ARC runner definition: l-x86iamx-44-450-h100-fab (2x NVIDIA H100, IMEX fabric) -# Scheduled on NodePool: p5-48xlarge (from nodepools-h100 module) -runner: - name: l-x86iamx-44-450-h100-fab - instance_type: p5.48xlarge - disk_size: 200 - vcpu: 44 - memory: 450Gi - gpu: 2 - imex_channels: 0 - # Dedicated fabric runner: fixed low cap, not scaled to the full reservation. - max_runners: 1 diff --git a/osdc/modules/arc-runners-h100/defs/l-x86iamx-88-900-h100-fab-4.yaml b/osdc/modules/arc-runners-h100/defs/l-x86iamx-88-900-h100-fab-4.yaml new file mode 100644 index 00000000..3b100f71 --- /dev/null +++ b/osdc/modules/arc-runners-h100/defs/l-x86iamx-88-900-h100-fab-4.yaml @@ -0,0 +1,14 @@ +# ARC runner definition: l-x86iamx-88-900-h100-fab-4 (4x NVIDIA H100) +# Scheduled on NodePool: p5-48xlarge (from nodepools-h100 module) +runner: + name: l-x86iamx-88-900-h100-fab-4 + instance_type: p5.48xlarge + disk_size: 200 + vcpu: 88 + memory: 900Gi + gpu: 4 + imex_channels: 0 + # Fixed-capacity cap: 1 reserved 8-GPU node / 4 GPUs per runner = 2 concurrent runners. + max_runners: + default: 2 + meta-prod-aws-uw1: 12 diff --git a/osdc/modules/arc-runners-h100/validation/README.md b/osdc/modules/arc-runners-h100/validation/README.md deleted file mode 100644 index 76c49f95..00000000 --- a/osdc/modules/arc-runners-h100/validation/README.md +++ /dev/null @@ -1,137 +0,0 @@ -# H100 CUDA fabric-handle IPC validation probe - -Rollout validation for the H100 fabric runner (`mt-l-x86iamx-44-450-h100-fab`) -and its IMEX channel. This is a manual probe, not a unit test. It is not wired -into `just test` or any module deploy. - -## What it validates - -That a torch workload on a fabric-enabled H100 runner actually uses CUDA -**fabric handles** for expandable-segments IPC, instead of silently falling back -to POSIX file-descriptor handles. - -## Why it exists (the false-green risk) - -`c10::cuda::get_fabric_access()` decides whether the CUDA caching allocator can -share expandable memory segments over a FABRIC handle. It probes NVML and then -runs a full CUDA allocate / export / import cycle over the fabric handle -(`isFabricSupported()` in `c10/cuda/PeerToPeerAccess.cpp`). That import is what -needs the IMEX channel device inside the container. - -If the IMEX channel is missing or unusable, the cycle fails and -`ExpandableSegment::detectHandleType()` (in `c10/cuda/CUDACachingAllocator.cpp`) -GRACEFULLY falls back to POSIX-fd handles. The workload still completes and exits -0. So a green job does not prove fabric was used - it can be a silent POSIX -fallback. This is the "false green" the rollout must guard against. - -There is no Python API that reports the handle type actually used, so the probe -captures the C++ INFO logs c10 emits and asserts on them: - -| C++ log line (INFO) | Meaning | -| ----------------------------------------------------- | ----------------------------- | -| `use fabric handle to share expandable segments` | fabric export was used (good) | -| `use posix fd to share expandable segments` | silent POSIX fallback (bad) | -| `use fabric handle to import expandable segments` | fabric import was used (good) | -| `using fabric to exchange memory handles` | `isFabricSupported()` passed | -| `... falling back to fd handle exchange` | fabric probe failed | - -The probe forces the expandable-segments IPC path (allocate a >2 MiB tensor, -`reduce_tensor` to export and cache the handle, then free it to run the segment -destructor - the path PR #190860 fixed), captures the logs, and prints: - -- `FABRIC_VALIDATION: PASS` - fabric SHARE line present, no POSIX-fallback line. -- `FABRIC_VALIDATION: FAIL` - fell back to POSIX, or the path was never exercised - (inconclusive), or the workload errored. Exit code is non-zero on FAIL. - -It deliberately leaves `expandable_segments_handle_type` UNSPECIFIED. UNSPECIFIED -is the auto-detect-or-fall-back path we must validate; forcing FABRIC would make -c10 hard-error instead of falling back, hiding the exact false-green under test. - -## How it works - -`fabric_probe.py` runs the torch workload in a child process and evaluates that -child's captured C++ logs after it exits (so glog buffers are flushed). - -- Multiprocess mode (default when >=2 GPUs) mirrors - `test/distributed/test_p2p_ipc.py::test_p2p_ipc_expandable_segments`: rank 0 on - GPU 0 allocates and exports; rank 1 on GPU 1 imports without pre-allocating - (the `#179220` regression shape). This exercises the real cross-GPU fabric - round trip the IMEX channel enables, asserting both the fabric SHARE and the - fabric IMPORT log lines. -- Single-process mode (`--single-process`, or automatic on a 1-GPU host) does - allocate / export / free in one process. Because `isFabricSupported()` already - runs a self-contained export+import cycle through the IMEX channel while - deciding the handle type, one GPU is enough to prove the channel is usable and - that the SHARE path took fabric. - -Required environment (set by the manifest, or `os.environ.setdefault` when run -standalone): `NVIDIA_IMEX_CHANNELS=0`, `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, -`TORCH_CUDA_EXPANDABLE_SEGMENTS_IPC=1`, `TORCH_CPP_LOG_LEVEL=INFO`, `GLOG_logtostderr=1`. - -## Image requirement - -The probe only validates the torch build it runs inside. That build MUST contain -the c10 expandable-segments fabric IPC code and be built with CUDA >= 12.4. -Prefer the exact image the H100 CI jobs run. If the build lacks the code, the -probe reports an inconclusive FAIL (no share log at all) rather than a false -pass - so a wrong image cannot masquerade as success. - -## Run it: (a) directly via kubectl on meta-prod-aws-uw1 - -From the `osdc/` repo root, against the `meta-prod-aws-uw1` cluster context: - -```bash -# 1. Edit fabric-validation-pod.yaml and set spec.containers[].image to the -# torch/CUDA image under validation (replaces REPLACE_WITH_TORCH_CUDA_IMAGE). - -# 2. Publish the probe source as a ConfigMap (single source of truth - the .py). -kubectl create configmap fabric-probe -n default \ - --from-file=fabric_probe.py=modules/arc-runners-h100/validation/fabric_probe.py - -# 3. Launch the probe Pod. -kubectl apply -f modules/arc-runners-h100/validation/fabric-validation-pod.yaml - -# 4. Watch the verdict (the last log line is FABRIC_VALIDATION: PASS/FAIL). -kubectl logs -f pod/fabric-validation -n default - -# 5. Clean up. -kubectl delete pod/fabric-validation configmap/fabric-probe -n default -``` - -A p5.48xlarge node must be available (Capacity Blocks reserved). The Pod -tolerates the H100 fleet taints and requests `nvidia.com/gpu: 2`. - -## Run it: (b) as a workflow_dispatch job on the fabric runner - -Running on the runner uses the CI job's own torch image, so it validates exactly -what production runs. No image placeholder to fill in. Example workflow: - -```yaml -name: h100-fabric-validation -on: - workflow_dispatch: -jobs: - fabric-probe: - runs-on: mt-l-x86iamx-44-450-h100-fab - steps: - - name: Checkout osdc (for the probe source) - uses: actions/checkout@v4 - - name: Run fabric validation probe - env: - NVIDIA_IMEX_CHANNELS: "0" - PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True" - TORCH_CUDA_EXPANDABLE_SEGMENTS_IPC: "1" - TORCH_CPP_LOG_LEVEL: "INFO" - GLOG_logtostderr: "1" - run: python modules/arc-runners-h100/validation/fabric_probe.py -``` - -The step fails (non-zero exit) if fabric was not used, turning the false-green -into a real red. Adjust the checkout to wherever the probe source is available -on the runner. - -## Files - -- `fabric_probe.py` - the probe (driver + torch workload + log assertion). -- `fabric-validation-pod.yaml` - the standalone kubectl Pod. -- `README.md` - this file. diff --git a/osdc/modules/arc-runners-h100/validation/fabric-validation-pod.yaml b/osdc/modules/arc-runners-h100/validation/fabric-validation-pod.yaml deleted file mode 100644 index d3ec40de..00000000 --- a/osdc/modules/arc-runners-h100/validation/fabric-validation-pod.yaml +++ /dev/null @@ -1,101 +0,0 @@ -# CUDA fabric-handle IPC validation Pod for the H100 fabric runner rollout. -# -# This is a manual, run-to-completion validation probe. It is NOT part of any -# module deploy and is NOT wired into `just`. Apply it by hand on the target -# cluster (meta-prod-aws-uw1), read the logs for the PASS/FAIL verdict, then -# delete it. See README.md in this directory. -# -# The probe source (fabric_probe.py) is mounted from a ConfigMap so it lives in -# exactly one place. Create the ConfigMap from the .py file before applying: -# -# kubectl create configmap fabric-probe -n default \ -# --from-file=fabric_probe.py=modules/arc-runners-h100/validation/fabric_probe.py -# kubectl apply -f modules/arc-runners-h100/validation/fabric-validation-pod.yaml -# kubectl logs -f pod/fabric-validation -n default -# kubectl delete pod/fabric-validation configmap/fabric-probe -n default -# -# IMAGE: set spec.containers[].image to the torch/CUDA build under validation. -# It MUST contain the c10 expandable-segments fabric IPC code and be built with -# CUDA >= 12.4. Prefer the exact image the H100 CI jobs run. If the build lacks -# the code the probe reports an inconclusive FAIL (no share log) rather than a -# false pass. See README.md. -apiVersion: v1 -kind: Pod -metadata: - name: fabric-validation - namespace: default - labels: - osdc.io/module: arc-runners-h100 - osdc.io/purpose: fabric-ipc-validation - annotations: - # Never let Karpenter disrupt the node mid-probe. - karpenter.sh/do-not-disrupt: "true" -spec: - restartPolicy: Never - # Pin to the p5.48xlarge H100 fleet (the fabric runner's node type). - nodeSelector: - node.kubernetes.io/instance-type: p5.48xlarge - # Tolerate every taint an H100 workflow node can carry so the probe schedules - # exactly where the fabric runner's job pods do. - tolerations: - - key: node-fleet - operator: Exists - effect: NoSchedule - - key: instance-type - operator: Exists - effect: NoSchedule - - key: nvidia.com/gpu - operator: Exists - effect: NoSchedule - - key: git-cache-not-ready - operator: Exists - effect: NoSchedule - containers: - - name: fabric-probe - # REPLACE with the torch/CUDA image under validation (see header + README). - image: REPLACE_WITH_TORCH_CUDA_IMAGE - command: ["python", "/probe/fabric_probe.py"] - env: - # Expose IMEX channel 0 to the container so cuMem*ShareableHandle with - # CU_MEM_HANDLE_TYPE_FABRIC can succeed. Without this the allocator - # silently falls back to POSIX fd and the probe fails. - - name: NVIDIA_IMEX_CHANNELS - value: "0" - # Drive the c10 allocator down the expandable-segments IPC path and turn - # on the C++ INFO logs the probe asserts against. Leave the handle type - # UNSPECIFIED so the auto-detect / fallback path is what gets validated. - - name: PYTORCH_CUDA_ALLOC_CONF - value: "expandable_segments:True" - - name: TORCH_CUDA_EXPANDABLE_SEGMENTS_IPC - value: "1" - - name: TORCH_CPP_LOG_LEVEL - value: "INFO" - - name: GLOG_logtostderr - value: "1" - resources: - requests: - cpu: "4" - memory: 32Gi - nvidia.com/gpu: "2" - limits: - cpu: "4" - memory: 32Gi - nvidia.com/gpu: "2" - volumeMounts: - - name: probe - mountPath: /probe - readOnly: true - # NCCL/torch multiprocessing want more than the default 64Mi /dev/shm. - - name: dshm - mountPath: /dev/shm - volumes: - - name: probe - configMap: - name: fabric-probe - items: - - key: fabric_probe.py - path: fabric_probe.py - - name: dshm - emptyDir: - medium: Memory - sizeLimit: 2Gi diff --git a/osdc/modules/arc-runners-h100/validation/fabric_probe.py b/osdc/modules/arc-runners-h100/validation/fabric_probe.py deleted file mode 100644 index a4915bf7..00000000 --- a/osdc/modules/arc-runners-h100/validation/fabric_probe.py +++ /dev/null @@ -1,293 +0,0 @@ -#!/usr/bin/env python3 -"""Rollout validation probe for CUDA fabric-handle IPC on H100 with an IMEX channel. - -This is NOT a unit test. It is a manual validation probe for the H100 fabric -runner rollout. It must never be wired into `just test`. - -Why this probe exists ---------------------- -`c10::cuda::get_fabric_access()` (c10/cuda/PeerToPeerAccess.cpp) probes NVML plus -a full CUDA allocate/export/import cycle over a FABRIC handle. When the IMEX -channel is missing or unusable that cycle fails and the caching allocator -(`ExpandableSegment::detectHandleType`, c10/cuda/CUDACachingAllocator.cpp) -GRACEFULLY falls back to POSIX-fd handles. A torch workload therefore exits 0 -whether or not fabric handles were ever used. Exit code alone cannot tell a -fabric run apart from a silent POSIX fallback ("false green"). - -There is no Python API that reports which handle type the allocator picked, so -this probe captures the C++ INFO logs emitted by c10 and ASSERTS on them: - - - "use fabric handle to share expandable segments" -> fabric export was used - - "use posix fd to share expandable segments" -> silent POSIX fallback - - "use fabric handle to import expandable segments" -> fabric import was used - -The probe forces the allocator down the expandable-segments IPC path (allocate ->2 MiB, share via reduce_tensor to cache the exported handle, then free to run -the segment destructor), captures the logs, and fails hard unless the fabric -SHARE line is present and no POSIX-fallback line is. - -Handle-type note: the probe deliberately leaves -`expandable_segments_handle_type` UNSPECIFIED (never forces FABRIC). UNSPECIFIED -is exactly the auto-detect-or-fall-back path we must validate; forcing FABRIC -would make c10 hard-error instead of falling back, which would hide the very -false-green this probe guards against. - -Modes ------ -- multiprocess (default when >=2 GPUs): mirrors test/distributed/test_p2p_ipc.py - test_p2p_ipc_expandable_segments. Rank 0 (GPU 0) allocates and exports; rank 1 - (GPU 1) imports without pre-allocating (the #179220 regression shape). This - exercises the real cross-GPU fabric round trip the IMEX channel enables. -- single-process (fallback / 1-GPU): allocate, reduce_tensor to export, free. - isFabricSupported() already runs a self-contained export+import cycle through - the IMEX channel while deciding the handle type, so even one GPU proves the - channel is usable and that the SHARE path took fabric. Use --single-process to - force it. - -The actual torch workload runs in a child process; its C++ logs are captured -after it fully exits (so glog buffers are flushed) and then evaluated. -""" - -from __future__ import annotations - -import argparse -import os -import subprocess -import sys - -# Set before any torch import so the CUDA allocator picks these up on first -# touch. setdefault lets the k8s manifest or an operator override any of them. -_ENV_DEFAULTS = { - "PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True", - "TORCH_CUDA_EXPANDABLE_SEGMENTS_IPC": "1", - "TORCH_CPP_LOG_LEVEL": "INFO", - "GLOG_logtostderr": "1", -} -for _k, _v in _ENV_DEFAULTS.items(): - os.environ.setdefault(_k, _v) - -# Contract with c10 source. These exact strings are emitted by -# c10/cuda/CUDACachingAllocator.cpp and c10/cuda/PeerToPeerAccess.cpp. -LOG_FABRIC_SHARE = "use fabric handle to share expandable segments" -LOG_POSIX_SHARE = "use posix fd to share expandable segments" -LOG_FABRIC_IMPORT = "use fabric handle to import expandable segments" -LOG_POSIX_IMPORT = "use posix fd to import expandable segments" -LOG_FABRIC_OK = "using fabric to exchange memory handles" -LOG_FABRIC_FALLBACK = "falling back to fd handle exchange" - -# float32 * 2Mi elements = 8 MiB; comfortably above the 2 MiB segment size so an -# expandable segment is created (mirrors test_p2p_ipc_expandable_segments). -TENSOR_ELEMENTS = 2 * 1024 * 1024 -MASTER_ADDR = "127.0.0.1" -MASTER_PORT = "29511" - -MODE_MARKER = "PROBE_MODE:" -VERDICT_MARKER = "FABRIC_VALIDATION:" - - -def _log(msg: str) -> None: - print(f"[probe] {msg}", flush=True) - - -def _single_process_workload() -> None: - import gc - - import torch - from torch.multiprocessing.reductions import reduce_tensor - - print(f"{MODE_MARKER} single-process", flush=True) - if not torch.cuda.is_available(): - raise SystemExit("CUDA is not available; cannot run the fabric probe") - - torch.cuda.set_device(0) - torch.cuda.memory._set_allocator_settings("expandable_segments:True") - torch.cuda.empty_cache() - - device = torch.device("cuda", 0) - mib = TENSOR_ELEMENTS * 4 / 1024 / 1024 - _log(f"allocating {mib:.0f} MiB expandable segment on {device}") - tensor = torch.randn(TENSOR_ELEMENTS, device=device) - - # reduce_tensor exports and caches the shareable handle, emitting the - # SHARE log line for the handle type the allocator selected. - meta = reduce_tensor(tensor) - _log("reduce_tensor() completed (handle exported)") - - # Free to run the ExpandableSegment destructor against the cached handle - # (the path PR #190860 fixed). - del meta - del tensor - gc.collect() - torch.cuda.empty_cache() - torch.cuda.synchronize() - _log("freed segment (destructor exercised)") - - -def _dist_worker(rank: int, world_size: int) -> None: - import gc - - import torch - import torch.distributed as dist - from torch.multiprocessing.reductions import reduce_tensor - - os.environ.setdefault("MASTER_ADDR", MASTER_ADDR) - os.environ.setdefault("MASTER_PORT", MASTER_PORT) - - torch.cuda.set_device(rank) - dist.init_process_group(backend="gloo", rank=rank, world_size=world_size) - try: - device = torch.device("cuda", rank) - if rank == 0: - torch.cuda.memory._set_allocator_settings("expandable_segments:True") - torch.cuda.empty_cache() - mib = TENSOR_ELEMENTS * 4 / 1024 / 1024 - _log(f"rank0 producer: allocating {mib:.0f} MiB expandable segment on {device}") - tensor = torch.randn(TENSOR_ELEMENTS, device=device) - meta = reduce_tensor(tensor) - _log("rank0 producer: exported handle, broadcasting to consumer") - dist.broadcast_object_list([meta], src=0) - dist.barrier() - del meta - del tensor - gc.collect() - torch.cuda.empty_cache() - _log("rank0 producer: freed segment (destructor exercised)") - else: - # Do NOT pre-allocate: a consumer that has never touched the - # allocator is the #179220 regression shape. - _log(f"rank{rank} consumer: receiving handle (no pre-allocation)") - recv: list[object] = [None] - dist.broadcast_object_list(recv, src=0) - func, args = recv[0] # type: ignore[misc] - args = list(args) - # args[6] is storage_device (rebuild_cuda_tensor): import onto this - # consumer's own GPU, exercising the cross-GPU fabric import. - args[6] = rank - tensor = func(*args) - _log(f"rank{rank} consumer: imported handle onto {device}") - dist.barrier() - del tensor - gc.collect() - torch.cuda.empty_cache() - _log(f"rank{rank} consumer: freed imported segment") - torch.cuda.synchronize() - finally: - dist.destroy_process_group() - - -def _multiprocess_workload() -> None: - import torch - import torch.multiprocessing as mp - - if not torch.cuda.is_available(): - raise SystemExit("CUDA is not available; cannot run the fabric probe") - device_count = torch.cuda.device_count() - if device_count < 2: - raise SystemExit(f"multiprocess mode needs >=2 GPUs, found {device_count}; use --single-process") - - world_size = 2 - print(f"{MODE_MARKER} multiprocess ({world_size} GPUs)", flush=True) - os.environ.setdefault("MASTER_ADDR", MASTER_ADDR) - os.environ.setdefault("MASTER_PORT", MASTER_PORT) - mp.spawn(_dist_worker, args=(world_size,), nprocs=world_size, join=True) - - -def _run_workload(single_process: bool) -> int: - import torch - - if single_process or torch.cuda.device_count() < 2: - _single_process_workload() - else: - _multiprocess_workload() - return 0 - - -def _verdict(log: str, workload_rc: int) -> tuple[bool, str]: - fabric_share = LOG_FABRIC_SHARE in log - posix_share = LOG_POSIX_SHARE in log - fabric_import = LOG_FABRIC_IMPORT in log - posix_import = LOG_POSIX_IMPORT in log - multiprocess = f"{MODE_MARKER} multiprocess" in log - - if workload_rc != 0: - return False, f"workload process exited non-zero ({workload_rc}); see logs above" - if not fabric_share and not posix_share: - return False, ( - "SHARE path never exercised: no fabric/posix share log seen. The torch build may " - "lack expandable-segments IPC, the segment was not created, or C++ INFO logging was " - "not enabled (TORCH_CPP_LOG_LEVEL=INFO). Result is inconclusive, not a pass." - ) - if posix_share: - return False, "fell back to POSIX fd on the SHARE (export) path - fabric was NOT used" - if multiprocess and posix_import: - return False, "fell back to POSIX fd on the IMPORT path - fabric was NOT used" - if multiprocess and not fabric_import: - return False, "consumer never imported via fabric; cross-GPU round trip did not complete" - return True, "fabric handle used on the expandable-segments IPC path" - - -def _evaluate(log: str, workload_rc: int) -> int: - signals = { - "fabric share": LOG_FABRIC_SHARE in log, - "posix share (fallback)": LOG_POSIX_SHARE in log, - "fabric import": LOG_FABRIC_IMPORT in log, - "posix import (fallback)": LOG_POSIX_IMPORT in log, - "isFabricSupported ok": LOG_FABRIC_OK in log, - "isFabricSupported fell back": LOG_FABRIC_FALLBACK in log, - } - _log("captured C++ log signals:") - for name, seen in signals.items(): - print(f" {'FOUND' if seen else '- '} {name}", flush=True) - - passed, reason = _verdict(log, workload_rc) - if passed: - print(f"{VERDICT_MARKER} PASS - {reason}", flush=True) - return 0 - print(f"{VERDICT_MARKER} FAIL - {reason}", flush=True) - return 1 - - -def _run_driver(single_process: bool) -> int: - cmd = [sys.executable, os.path.abspath(__file__), "--run"] - if single_process: - cmd.append("--single-process") - _log("launching workload: " + " ".join(cmd)) - - proc = subprocess.Popen( - cmd, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - text=True, - bufsize=1, - ) - captured: list[str] = [] - if proc.stdout is not None: - for line in proc.stdout: - sys.stdout.write(line) - sys.stdout.flush() - captured.append(line) - workload_rc = proc.wait() - return _evaluate("".join(captured), workload_rc) - - -def main() -> int: - parser = argparse.ArgumentParser(description=__doc__) - parser.add_argument( - "--run", - action="store_true", - help="internal: execute the torch workload (invoked by the driver)", - ) - parser.add_argument( - "--single-process", - action="store_true", - help="force the single-process, single-GPU workload", - ) - args = parser.parse_args() - - if args.run: - return _run_workload(single_process=args.single_process) - return _run_driver(single_process=args.single_process) - - -if __name__ == "__main__": - sys.exit(main()) From 0aa8a83893026dfc00ce1f5d7ed817fa6d49a242 Mon Sep 17 00:00:00 2001 From: Jean Schmidt Date: Fri, 24 Jul 2026 16:42:45 -0700 Subject: [PATCH 4/4] Retry IMEX char-major lookup on first boot - Poll /proc/devices for nvidia-caps-imex-channels up to 5 times, 2s apart - Break as soon as the char major appears; log each retry attempt to stderr - Update the absent-class error to note retries were exhausted nvidia.ko may not have registered the nvidia-caps-imex-channels char class in /proc/devices by the time the IMEX helper first runs at boot, causing the channel node creation to fail spuriously. Retrying gives the driver time to register the dynamically-assigned major before giving up. Signed-off-by: Jean Schmidt --- .../nodepools-h100/scripts/h100-node-setup.sh | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh b/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh index a117454f..7b26249f 100755 --- a/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh +++ b/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh @@ -163,9 +163,21 @@ if [ -e "$CHANNEL" ]; then echo "[h100-node-setup] IMEX $CHANNEL already present" exit 0 fi -MAJOR=$(awk '$2=="nvidia-caps-imex-channels"{print $1}' /proc/devices) +# nvidia.ko may not have registered the nvidia-caps-imex-channels char class in +# /proc/devices yet on first boot; retry until it appears. +IMEX_MAJOR_RETRIES=5 +IMEX_MAJOR_RETRY_SLEEP=2 +MAJOR="" +for attempt in $(seq 1 "$IMEX_MAJOR_RETRIES"); do + MAJOR=$(awk '$2=="nvidia-caps-imex-channels"{print $1}' /proc/devices) + if [ -n "$MAJOR" ]; then + break + fi + echo "[h100-node-setup] nvidia-caps-imex-channels not yet in /proc/devices (attempt ${attempt}/${IMEX_MAJOR_RETRIES}); retrying in ${IMEX_MAJOR_RETRY_SLEEP}s" >&2 + sleep "$IMEX_MAJOR_RETRY_SLEEP" +done if [ -z "$MAJOR" ]; then - echo "ERROR: [h100-node-setup] char class nvidia-caps-imex-channels absent from /proc/devices; cannot create $CHANNEL" >&2 + echo "ERROR: [h100-node-setup] char class nvidia-caps-imex-channels absent from /proc/devices after retries; cannot create $CHANNEL" >&2 exit 1 fi if mkdir -p /dev/nvidia-caps-imex-channels \