diff --git a/osdc/CLAUDE.md b/osdc/CLAUDE.md index 89a19484..b1cdd7b1 100644 --- a/osdc/CLAUDE.md +++ b/osdc/CLAUDE.md @@ -79,3 +79,4 @@ Reference documentation in `docs/`: | `docs/node-warmup-and-scheduling-gates.md` | Full node initialization sequence — taints, DaemonSets, init containers before job scheduling | | `docs/arc-fork-build-deploy.md` | ARC fork (jeanschmidt/actions-runner-controller) build/release workflow and chart publishing | | `docs/pypi-package-cache.md` | PyPI wheel cache architecture, slug naming, S3 layout, runner integration | +| `docs/h100-fabric-handles-imex-channels.md` | Enabling CUDA fabric handles (`CU_MEM_HANDLE_TYPE_FABRIC`) on H100 runners — IMEX channel creation, the gated fabric runner, and verification | diff --git a/osdc/docs/h100-fabric-handles-imex-channels.md b/osdc/docs/h100-fabric-handles-imex-channels.md new file mode 100644 index 00000000..f3836d31 --- /dev/null +++ b/osdc/docs/h100-fabric-handles-imex-channels.md @@ -0,0 +1,228 @@ +# H100 Fabric Handles and IMEX Channels (OSDC) + +## Context + +PyTorch [pytorch/pytorch#190860](https://github.com/pytorch/pytorch/pull/190860) fixed a +destructor crash in `ExpandableSegment::unmapHandles` — `std::bad_variant_access` when the +cached `shareable_handle` holds a `CUmemFabricHandle` instead of a POSIX fd. A follow-up +question asked whether OSDC could run CI that exercises the CUDA **fabric handle** path, and +whether that requires new (multi-node NVLink / GB200) hardware. An automated bot comment on +the PR asserted that standard AWS `p5` (H100) instances report GPU fabric state +`NOT_SUPPORTED`, and that fabric CI would need fabric-provisioned hardware plus the +`nvidia-imex` daemon. + +This doc records what is actually true on the OSDC H100 fleet, verified against a live node +on 2026-07-24. + +## TL;DR + +- **OSDC H100 (`p5.48xlarge`) nodes are fabric-capable today.** `CU_MEM_HANDLE_TYPE_FABRIC` + allocate → export → import → free was verified working on a live node. **No new hardware, + no GB200 NVL72, no `nvidia-imex` daemon.** +- **The full production injection path is verified end-to-end** — a host-created channel plus + `NVIDIA_IMEX_CHANNELS` on an *unprivileged* pod: the pod starts and the fabric probe passes. +- The bot's "p5 reports `NOT_SUPPORTED`" claim is **false** — a live p5 reports fabric + `State: Completed / Status: Success`. +- The **only** missing piece is the **IMEX channel device** + (`/dev/nvidia-caps-imex-channels/channel0`), which is absent on our nodes by design. It is + a per-node/per-container config gap, not a hardware gap. + +## Background: fd vs fabric handles + +When a CUDA process shares a VMM allocation with another process, the exported handle is one +of two kinds: +- **POSIX fd** (`CU_MEM_HANDLE_TYPE_POSIX_FILE_DESCRIPTOR`) — works within a single machine. + This is what every OSDC runner uses today. +- **Fabric** (`CU_MEM_HANDLE_TYPE_FABRIC`, a `CUmemFabricHandle`, CUDA 12.4+) — rides the + NVLink fabric. PyTorch's expandable-segments IPC / SymmetricMemory selects this when + `get_fabric_access()` is true. + +The #190860 crash only occurs on the fabric branch: `unmapHandles` did +`close(std::get(*h.shareable_handle))`, which throws from a destructor when the variant +holds a fabric handle (fixed by switching to the non-throwing `std::get_if`). + +## Finding: verified on a live node (2026-07-24) + +Node `ip-10-8-105-26` (`p5.48xlarge`, 8× H100 80GB) on cluster `meta-prod-aws-uw1`, driver +`580.159.03`, CUDA `13.0`: + +- `nvidia-smi -q` GPU Fabric: `State: Completed`, `Status: Success`, `CliqueId: 0`, all-zero + `ClusterUUID` (a standalone single-node fabric island — not part of a multi-node clique). +- With an IMEX channel device present (`mknod .../channel0 c 242 0`, mode `0666`), a minimal + CUDA driver-API program ran the full cycle successfully: + `cuMemCreate(requestedHandleTypes=CU_MEM_HANDLE_TYPE_FABRIC)` → + `cuMemExportToShareableHandle(FABRIC)` → `cuMemImportFromShareableHandle(FABRIC)` → + `cuMemRelease` → exit 0 (`FABRIC_PROBE: PASS`). +- `CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_FABRIC_SUPPORTED` returned 1 (informational only — + documented-unreliable; the export/import round-trip is the authoritative test). + +### Requirements for single-node fabric handles on H100 +1. **NVSwitch node** — `p5` (H100) and `p6-b200` (B200) qualify; each is an 8-GPU intra-node + NVSwitch island. +2. **Fabric Manager running** — `nv-fabricmanager`; already running via the EKS NVIDIA AMI, + which is why fabric `State` is `Completed`. FM configures NVSwitch/fabric; it does **not** + create IMEX channels. +3. **IMEX channel device** — `/dev/nvidia-caps-imex-channels/channelN` (char major 242 on + this AMI), created via `mknod` or the `NVreg_CreateImexChannel0=1` driver param, and + accessible (`0666`) to the process. Required even single-node: the driver enforces channel + access on `cuMemImportFromShareableHandle(FABRIC)` (else `CUDA_ERROR_NOT_PERMITTED`). +4. **NOT required single-node:** the `nvidia-imex` daemon (it only coordinates cross-node / + multi-node NVLink domains). + +### Current state on OSDC H100 nodes +- Fabric Manager: running. Fabric state: `Completed`. +- IMEX channel device: **absent** (not created by default; the node bootstrap deliberately + does not create it). +- Net effect: `c10::cuda::get_fabric_access()` (PyTorch `c10/cuda/PeerToPeerAccess.cpp:164`) + returns **false** — its `isFabricSupported()` probe's `cuMemImportFromShareableHandle(FABRIC)` + fails with `CUDA_ERROR_NOT_PERMITTED` when no channel is accessible — so the allocator uses + the POSIX-fd path. That is why no OSDC CI has ever hit the #190860 fabric-destructor bug. + +## The documented Fabric Manager / IMEX race + +`modules/nodepools-h100/scripts/h100-node-setup.sh:122-143` (identical block in +`modules/nodepools-b200/scripts/b200-node-setup.sh`), added by osdc#552 (2026-05-12) after +osdc#541's eager approach caused a 100% H100 node-kill rate: + +- **The race:** starting `nvidia-fabricmanager` from cloud-init races the GPUs' on-die + "local FM" init. Host FM finishes NVSwitch routing in seconds, then waits up to the 30 s + GFM Wait Timeout for the GPU local-FM instances to register over NVLink Inband. From + cloud-init they are not ready yet → FM aborts with "config error type 8 / not all local + fabric manager instances finished their configuration" → the hard-failing setup script + fails → the node never joins → Karpenter reaps it. +- **The fix:** do nothing in cloud-init. The AMI already `systemctl enable`d FM, so systemd + starts it later in normal boot (GPUs ready), completing in <1 s. +- **Scope:** the race is specific to **starting the FM daemon early**. Creating an IMEX + channel device does not start FM and does not re-trigger this race. + +**Correction to the script comment.** The comment states "IMEX channels are only needed for +multi-node NVLink." That is true for **NCCL** (intra-node NCCL/NVLS multicast uses POSIX-fd +handles) but **not** for CUDA `CU_MEM_HANDLE_TYPE_FABRIC` (PyTorch expandable-segments IPC / +SymmetricMemory), which needs the channel even on a single node. The comment also correctly +notes that `modprobe nvidia-caps-imex-channels` always fails — IMEX is compiled into +`nvidia.ko` (there is no separate `.ko`); the channel is created via `mknod` or the +`NVreg_CreateImexChannel0` module param, not by loading a module. + +## Enabling fabric handles on the existing fleet (design) + +No new hardware required. Two pieces: + +### 1. Create the channel on the node +- **Runtime `mknod`** (fits ephemeral Karpenter nodes): + `mknod /dev/nvidia-caps-imex-channels/channel0 c 0 && chmod 0666`, deriving + `` from `/proc/devices` (242 on the current AMI). Add to `h100-node-setup.sh` after + the `nvidia-smi -pm 1` line (:120). The script runs after the driver is loaded, so a + `modprobe.d` drop-in would not take effect until reboot — hence runtime `mknod`, not + modprobe, at cloud-init time. +- **Reboot-persistent alternative:** `options nvidia NVreg_CreateImexChannel0=1` in + `/etc/modprobe.d/` + initramfs rebuild (auto-creates a world-accessible channel0 on every + driver load). Heavier; for a baked AMI or fleet-wide rollout. +- Neither starts Fabric Manager, so neither re-triggers the documented race. + +### 2. Inject the channel into runner job pods +- OSDC uses the **stock** `k8s-device-plugin` (`base/kubernetes/nvidia-device-plugin.yaml:45-48`, + envvar strategy) with the NVIDIA container-toolkit runtime hook. Node toolkit config + (`/etc/nvidia-container-runtime/config.toml`) has + `accept-nvidia-visible-devices-envvar-when-unprivileged = true` (default) and CDI enabled, + so **unprivileged** pods can request IMEX channels via env. +- Set `NVIDIA_IMEX_CHANNELS="0"` on the job container (the toolkit maps it to CDI + `runtime.nvidia.com/imex-channel=0`). Equivalent alternatives: CDI device + `nvidia.com/imex-channel=0`, or the `/dev/null → /var/run/nvidia-container-devices/imex/channel0` + volume-mount. +- **Wire-up point:** the job-container env block in + `modules/arc-runners/templates/runner.yaml.tpl:431-464`, driven by a new per-def field + (e.g. `imex_channels`) read in `modules/arc-runners/scripts/python/generate_runners.py` + (~:242, replacements dict ~:429-467) → a new `{{IMEX_ENV}}` placeholder. Set the field only + on `modules/arc-runners-h100/defs/*.yaml` to gate to H100. + +### FAILS CLOSED — ordering matters +If `NVIDIA_IMEX_CHANNELS` is set on a pod but the host channel does not exist, the toolkit +stat's the host device, fails, and the pod goes to **`StartError`**: +``` +failed to apply edits for device "runtime.nvidia.com/imex-channel=0": +failed to stat CDI host device "/dev/nvidia-caps-imex-channels/channel0": no such file or directory +``` +So channel creation on the node MUST land before/with the env injection, or **every** H100 +runner pod fails to start. Conversely, creating the host channel is harmless to pods that do +NOT request it: injection is per-container, so their `get_fabric_access()` stays false and +their behavior is unchanged. This is what makes per-def gating safe. + +## How this was verified (2026-07-24) + +- A **self-contained privileged pod requesting all 8 GPUs** so it owns a whole node (no CI + co-location). A 1-GPU request instead co-locates onto a busy CI node — avoid. An 8-GPU + request routes to the `p5-large` Karpenter pool and scales a fresh `reserved` node. +- Image `nvcr.io/nvidia/cuda:13.0.0-devel-ubuntu24.04` (pulls via the node's Harbor + `nvcr.io` mirror; `-devel` ships `nvcc`). +- The pod created the channel in-container (`mknod`), compiled a driver-API probe + (`nvcc ... -L/usr/local/cuda/lib64/stubs -lcuda`), and ran it → `FABRIC_PROBE: PASS`. +- Isolation / scheduling facts: the `p5-48xlarge` pool is `consolidationPolicy: WhenEmpty`, + `consolidateAfter: 1h`; fresh p5 nodes carry a `git-cache-not-ready:NoSchedule` startup + taint (must be tolerated); a pod holding a non-DaemonSet workload pins its node against + consolidation. + +### Production injection path — verified (2026-07-24) +The unprivileged toolkit-injection path was confirmed end-to-end on an isolated `p5-large` +node (`ip-10-8-94-202`) with a single pod: a **privileged init container** `mknod`'d the host +channel (`crw-rw-rw- 242,0 channel0`), then the **unprivileged main container** +(`NVIDIA_IMEX_CHANNELS=0`, no privileged securityContext) **started** — no `StartError`, in +contrast to the bare attempt with no host channel — with +`/dev/nvidia-caps-imex-channels/channel0` injected by the toolkit, and the fabric probe +returned `FABRIC_PROBE: PASS`. This is the exact production wiring: node bootstrap creates the +host channel, the toolkit injects it into the unprivileged runner pod via the env, and the +unprivileged process uses fabric handles successfully. Caveat: the test container ran as +uid 0 (root) but *unprivileged* (no privileged securityContext); the channel's `0666` mode +makes it accessible to non-root runner UIDs (e.g. uid 1000) as well. + +## Hardware scope +- **H100 (`p5.48xlarge`): verified.** +- **B200 (`p6-b200.48xlarge`): expected identical** (same NVSwitch single-node architecture, + same AMI and node bootstrap) but not yet tested. +- **GB200 NVL72 (cross-node MNNVL):** a different regime that *does* need the `nvidia-imex` + daemon; not present in OSDC and not required for single-node fabric handles. On AWS this is + P6e-GB200 UltraServers (Capacity Blocks only) — overkill for exercising this code path. + +## Key file pointers + +OSDC (`ci-infra/osdc`): +- `modules/nodepools-h100/scripts/h100-node-setup.sh:122-143` — FM/IMEX "do nothing" + rationale; `nvidia-smi -pm 1` at :120. +- `modules/arc-runners/templates/runner.yaml.tpl:431-464` — job-container env block + (injection point). +- `modules/arc-runners/scripts/python/generate_runners.py` — ~:242 (read def field), + :359-378 (GPU tolerations/affinity/requests logic), ~:429-467 (replacements dict). +- `base/kubernetes/nvidia-device-plugin.yaml:45-48` — stock device plugin, no IMEX config. +- `modules/nodepools-h100/defs/{p5,p5-large}.yaml`, `modules/arc-runners-h100/defs/*.yaml`. + +PyTorch (`pytorch/pytorch`): +- `c10/cuda/PeerToPeerAccess.cpp:164` (`get_fabric_access`), :110 (`isFabricSupported`). +- `c10/cuda/CUDACachingAllocator.cpp:826` (`detectHandleType`), :620-638 (`share`), :942 + (destructor `std::get` fixed by #190860), :993 (`shareable_handle` variant). +- `test/distributed/test_p2p_ipc.py::test_p2p_ipc_expandable_segments` (currently + `@unittest.skip`, #189879). +- PR: https://github.com/pytorch/pytorch/pull/190860 + +## Sources (NVIDIA) +- CUDA VMM — fabric handles single- and multi-node: + https://docs.nvidia.com/cuda/cuda-programming-guide/04-special-topics/virtual-memory-management.html +- CUDA Driver API — `CUDA_ERROR_NOT_PERMITTED` on fabric import: + https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__VA.html +- IMEX channels guide: https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/imexchannels.html +- IMEX overview (daemon = multi-node only): + https://docs.nvidia.com/multi-node-nvlink-systems/imex-guide/overview.html +- Fabric Manager user guide (single-node HGX/DGX scope): + https://docs.nvidia.com/datacenter/tesla/fabric-manager-user-guide/index.html +- NVML fabric-state defs: https://docs.nvidia.com/deploy/nvml-api/group__nvmlFabricDefs.html +- Container Toolkit CDI / IMEX: + https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html +- Single-node fabric-handle confirmation (NVIDIA staff, dev forum): + https://forums.developer.nvidia.com/t/cudevicegetattribute-shows-i-can-use-fabric-handle-but-actually-i-cannot/336426 + +## Source / provenance +Verified on `meta-prod-aws-uw1` (`p5.48xlarge`, driver 580.159.03, CUDA 13.0) on 2026-07-24 +via two isolated probe pods: a self-contained privileged pod that created the channel +in-container and ran the fabric probe (node `ip-10-8-105-26`), and a two-container pod that +created the channel on the host and ran the probe from an unprivileged container with the +channel toolkit-injected via `NVIDIA_IMEX_CHANNELS` (node `ip-10-8-94-202`) — both +`FABRIC_PROBE: PASS`. Plus source review of OSDC node/runner modules and PyTorch `c10/cuda`. +Prompted by pytorch/pytorch#190860. diff --git a/osdc/modules/arc-runners-h100/defs/l-bx86iamx-176-1800-h100-fab-8.yaml b/osdc/modules/arc-runners-h100/defs/l-bx86iamx-176-1800-h100-fab-8.yaml new file mode 100644 index 00000000..e9bbb539 --- /dev/null +++ b/osdc/modules/arc-runners-h100/defs/l-bx86iamx-176-1800-h100-fab-8.yaml @@ -0,0 +1,19 @@ +# ARC runner definition: l-bx86iamx-176-1800-h100-fab-8 (8x NVIDIA H100, full node) +# Scheduled on NodePool fleet: p5-large (from nodepools-h100 module). +# Uses its OWN fleet — NOT the shared p5-48xlarge pool — because that pool pins +# topologyManagerPolicy: single-numa-node, which an all-8-GPU pod can never +# satisfy on a 2-NUMA-node p5.48xlarge (TopologyAffinityError). See +# nodepools-h100/defs/p5-large.yaml for the full rationale. +runner: + name: l-bx86iamx-176-1800-h100-fab-8 + instance_type: p5.48xlarge + node_fleet: p5-large + disk_size: 200 + vcpu: 176 + memory: 1800Gi + gpu: 8 + imex_channels: 0 + # Fixed-capacity cap: 1 reserved 8-GPU node / 8 GPUs per runner = 1 concurrent runner. + max_runners: + default: 1 + meta-prod-aws-uw1: 6 diff --git a/osdc/modules/arc-runners-h100/defs/l-x86iamx-22-225-h100-fab.yaml b/osdc/modules/arc-runners-h100/defs/l-x86iamx-22-225-h100-fab.yaml new file mode 100644 index 00000000..f9fd8b0e --- /dev/null +++ b/osdc/modules/arc-runners-h100/defs/l-x86iamx-22-225-h100-fab.yaml @@ -0,0 +1,15 @@ +# ARC runner definition: l-x86iamx-22-225-h100-fab (1x NVIDIA H100) +# Scheduled on NodePool: p5-48xlarge (from nodepools-h100 module) +# Resource math: 176 usable vCPU / 8 GPUs = 22 vCPU/GPU, 1800 GiB / 8 = 225 GiB/GPU +runner: + name: l-x86iamx-22-225-h100-fab + instance_type: p5.48xlarge + disk_size: 200 + vcpu: 22 + memory: 225Gi + gpu: 1 + imex_channels: 0 + # Fixed-capacity cap: 1 reserved 8-GPU node / 1 GPU per runner = 8 concurrent runners. + max_runners: + default: 8 + meta-prod-aws-uw1: 48 diff --git a/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab-2.yaml b/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab-2.yaml new file mode 100644 index 00000000..9b762183 --- /dev/null +++ b/osdc/modules/arc-runners-h100/defs/l-x86iamx-44-450-h100-fab-2.yaml @@ -0,0 +1,14 @@ +# ARC runner definition: l-x86iamx-44-450-h100-fab-2 (2x NVIDIA H100) +# Scheduled on NodePool: p5-48xlarge (from nodepools-h100 module) +runner: + name: l-x86iamx-44-450-h100-fab-2 + instance_type: p5.48xlarge + disk_size: 200 + vcpu: 44 + memory: 450Gi + gpu: 2 + imex_channels: 0 + # Fixed-capacity cap: 1 reserved 8-GPU node / 2 GPUs per runner = 4 concurrent runners. + max_runners: + default: 4 + meta-prod-aws-uw1: 24 diff --git a/osdc/modules/arc-runners-h100/defs/l-x86iamx-88-900-h100-fab-4.yaml b/osdc/modules/arc-runners-h100/defs/l-x86iamx-88-900-h100-fab-4.yaml new file mode 100644 index 00000000..3b100f71 --- /dev/null +++ b/osdc/modules/arc-runners-h100/defs/l-x86iamx-88-900-h100-fab-4.yaml @@ -0,0 +1,14 @@ +# ARC runner definition: l-x86iamx-88-900-h100-fab-4 (4x NVIDIA H100) +# Scheduled on NodePool: p5-48xlarge (from nodepools-h100 module) +runner: + name: l-x86iamx-88-900-h100-fab-4 + instance_type: p5.48xlarge + disk_size: 200 + vcpu: 88 + memory: 900Gi + gpu: 4 + imex_channels: 0 + # Fixed-capacity cap: 1 reserved 8-GPU node / 4 GPUs per runner = 2 concurrent runners. + max_runners: + default: 2 + meta-prod-aws-uw1: 12 diff --git a/osdc/modules/arc-runners/scripts/python/generate_runners.py b/osdc/modules/arc-runners/scripts/python/generate_runners.py index c51e187f..cf48d720 100755 --- a/osdc/modules/arc-runners/scripts/python/generate_runners.py +++ b/osdc/modules/arc-runners/scripts/python/generate_runners.py @@ -251,6 +251,7 @@ def generate_runner( max_burst_capacity = runner.get("max_burst_capacity", 0) hud_failure_base_capacity = runner.get("hud_failure_base_capacity", 0) node_fleet_override = runner.get("node_fleet") + imex_channels = runner.get("imex_channels") proactive_cap = cluster_config.get("proactive_capacity_max") if proactive_cap is not None: proactive_capacity = min(proactive_capacity, proactive_cap) @@ -377,6 +378,14 @@ def generate_runner( gpu_request = "" gpu_limit = "" + # Optional NVIDIA_IMEX_CHANNELS env on the workflow job container. Trails the + # preceding env value line inline so an unset field leaves that line — and the + # whole rendered runner — byte-identical to output without this feature. + if imex_channels is not None: + imex_env = f'\n - name: NVIDIA_IMEX_CHANNELS\n value: "{imex_channels}"' + else: + imex_env = "" + # Runner class isolation snippets (workflow pod only) # Release runners: required affinity to target release nodes # Regular runners: required affinity to avoid release nodes (DoesNotExist) @@ -442,6 +451,7 @@ def generate_runner( "{{GPU_NODE_SELECTOR_AFFINITY}}": gpu_node_selector_affinity, "{{GPU_REQUEST}}": gpu_request, "{{GPU_LIMIT}}": gpu_limit, + "{{IMEX_ENV}}": imex_env, "{{MODULE_NAME}}": module_name, "{{RUNNER_IMAGE}}": runner_image, # HF cache write target (used by the ci-refresh-hf-cache workflow's OIDC diff --git a/osdc/modules/arc-runners/scripts/python/test_generate_runners.py b/osdc/modules/arc-runners/scripts/python/test_generate_runners.py index d1ede96a..ae476e7f 100644 --- a/osdc/modules/arc-runners/scripts/python/test_generate_runners.py +++ b/osdc/modules/arc-runners/scripts/python/test_generate_runners.py @@ -200,6 +200,7 @@ def make_def_file( node_fleet=None, scheduler_name=None, fresh_multiplier=None, + imex_channels=None, ): """Write a runner def YAML and return the path. @@ -232,6 +233,8 @@ def make_def_file( runner["scheduler_name"] = scheduler_name if fresh_multiplier is not None: runner["fresh_multiplier"] = fresh_multiplier + if imex_channels is not None: + runner["imex_channels"] = imex_channels content = {"runner": runner} p = tmp_path / f"{name}.yaml" p.write_text(yaml.dump(content, default_flow_style=False)) @@ -2917,6 +2920,49 @@ def test_main_keeps_when_cluster_has_pypi_cache_module(self, tmp_path, monkeypat assert "# END_PYPI_CACHE" not in content +class TestImexChannels: + """Optional NVIDIA_IMEX_CHANNELS env var on the workflow job container.""" + + def _cluster_config(self): + return { + "github_config_url": "url", + "github_secret_name": "secret", + "runner_name_prefix": "", + } + + def test_imex_channels_set_injects_env(self, tmp_path, real_template): + """imex_channels: 0 renders NVIDIA_IMEX_CHANNELS="0" on the job container. + + 0 is deliberate: it is falsy, so this also proves the gate uses + ``is not None`` rather than truthiness (a truthiness check would drop 0). + """ + def_file = make_def_file( + tmp_path, "imex-runner", "p6-b200.48xlarge", 22, 225, gpu=1, disk_size=600, imex_channels=0 + ) + output_dir = tmp_path / "out" + output_dir.mkdir() + + assert generate_runner(def_file, real_template, self._cluster_config(), output_dir, "arc-runners") is True + + docs = list(yaml.safe_load_all((output_dir / "imex-runner.yaml").read_text())) + cm_data = yaml.safe_load(docs[1]["data"]["job-pod.yaml"]) + env_vars = {e["name"]: e["value"] for e in cm_data["spec"]["containers"][0]["env"]} + assert env_vars["NVIDIA_IMEX_CHANNELS"] == "0" + + def test_imex_channels_unset_omits_env(self, tmp_path, real_template): + """Without imex_channels, NVIDIA_IMEX_CHANNELS is absent from the job container.""" + def_file = make_def_file(tmp_path, "plain-runner", "c7i.24xlarge", 4, 16) + output_dir = tmp_path / "out" + output_dir.mkdir() + + assert generate_runner(def_file, real_template, self._cluster_config(), output_dir, "arc-runners") is True + + docs = list(yaml.safe_load_all((output_dir / "plain-runner.yaml").read_text())) + cm_data = yaml.safe_load(docs[1]["data"]["job-pod.yaml"]) + env_names = [e["name"] for e in cm_data["spec"]["containers"][0]["env"]] + assert "NVIDIA_IMEX_CHANNELS" not in env_names + + def _fresh_multiplier_env_value(output_dir, runner_name): docs = list(yaml.safe_load_all((output_dir / f"{runner_name}.yaml").read_text())) helm = docs[0] diff --git a/osdc/modules/arc-runners/templates/runner.yaml.tpl b/osdc/modules/arc-runners/templates/runner.yaml.tpl index aadd18ff..824d3b38 100644 --- a/osdc/modules/arc-runners/templates/runner.yaml.tpl +++ b/osdc/modules/arc-runners/templates/runner.yaml.tpl @@ -461,7 +461,7 @@ data: value: "{{HF_CACHE_REGION}}" # END_HF_CACHE - name: TORCH_CI_MAX_MEMORY - value: "{{MEMORY_BYTES}}" + value: "{{MEMORY_BYTES}}"{{IMEX_ENV}} # Workflow container gets the actual compute resources resources: requests: diff --git a/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh b/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh index 56380cae..7b26249f 100755 --- a/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh +++ b/osdc/modules/nodepools-h100/scripts/h100-node-setup.sh @@ -119,8 +119,8 @@ systemctl enable cpu-performance.service # Enable GPU persistence mode for consistent performance nvidia-smi -pm 1 || true -# ---- Fabric Manager / IMEX: do nothing here ---- -# Background for future maintainers: +# ---- Fabric Manager / IMEX ---- +# Fabric Manager daemon: deliberately left untouched here. # * The AMI (amazon-eks-node-al2023-x86_64-nvidia-*) already installs and # enables nvidia-fabricmanager and nvidia-persistenced. systemd brings FM # up later in boot, after the GPU-resident "local FM" instances have @@ -131,15 +131,93 @@ nvidia-smi -pm 1 || true # NVLink Inband. During cloud-init they are usually not ready yet, so FM # exits with "config error type 8 / not all local fabric manager # instances finished their configuration", taking the whole node out. -# * IMEX support on this AMI is compiled into nvidia.ko (char device class -# nvidia-caps-imex-channels, major 242). There is no separate -# nvidia-caps-imex-channels.ko; `modprobe nvidia-caps-imex-channels` -# will always fail. -# * Single-node NVLS multicast on p5 does not require an IMEX channel -# device - NCCL uses POSIX FD handles intranode. IMEX channels are only -# needed for multi-node NVLink (MNNVL / GB200 UltraServers). -# If a fresh provision ever shows FM failing on the normal boot path (not -# cloud-init), prefer a systemd drop-in with After=nvidia-persistenced.service -# and Restart=on-failure over taking control of FM here. +# If a fresh provision ever shows FM failing on the normal boot path (not +# cloud-init), prefer a systemd drop-in with After=nvidia-persistenced.service +# and Restart=on-failure over taking control of FM here. +# +# IMEX channel device: created below, for the current boot and (via an enabled +# oneshot) across reboots. +# * IMEX is compiled into nvidia.ko (char device class +# nvidia-caps-imex-channels; its char major is assigned dynamically, so it +# is read from /proc/devices rather than hardcoded). There is no separate +# nvidia-caps-imex-channels.ko - `modprobe nvidia-caps-imex-channels` always +# fails. +# * The driver does not auto-create the channel node, yet a single-node CUDA +# VMM fabric handle (CU_MEM_HANDLE_TYPE_FABRIC) - used by PyTorch +# expandable-segments IPC - needs /dev/nvidia-caps-imex-channels/channel0 to +# exist even on one host. NCCL's intra-node path uses POSIX FD handles and +# does not need a channel; the CUDA VMM fabric-handle path does, even on a +# single host. Fabric Manager is not required for this - the channel node +# plus the driver's single-node fabric are sufficient. +# +# The creation logic is written to a helper so the current boot and the oneshot +# share one implementation. It cannot be an inline ExecStart=/bin/bash -c with +# awk: systemd expands every unescaped `$` in ExecStart itself (undefined -> +# empty), so awk field refs and `$(...)` would be mangled. A plain script path +# has nothing for systemd to expand. +IMEX_HELPER=/usr/local/sbin/nvidia-imex-channel.sh +cat >"$IMEX_HELPER" <<'EOFHELPER' +#!/bin/bash +CHANNEL=/dev/nvidia-caps-imex-channels/channel0 +if [ -e "$CHANNEL" ]; then + echo "[h100-node-setup] IMEX $CHANNEL already present" + exit 0 +fi +# nvidia.ko may not have registered the nvidia-caps-imex-channels char class in +# /proc/devices yet on first boot; retry until it appears. +IMEX_MAJOR_RETRIES=5 +IMEX_MAJOR_RETRY_SLEEP=2 +MAJOR="" +for attempt in $(seq 1 "$IMEX_MAJOR_RETRIES"); do + MAJOR=$(awk '$2=="nvidia-caps-imex-channels"{print $1}' /proc/devices) + if [ -n "$MAJOR" ]; then + break + fi + echo "[h100-node-setup] nvidia-caps-imex-channels not yet in /proc/devices (attempt ${attempt}/${IMEX_MAJOR_RETRIES}); retrying in ${IMEX_MAJOR_RETRY_SLEEP}s" >&2 + sleep "$IMEX_MAJOR_RETRY_SLEEP" +done +if [ -z "$MAJOR" ]; then + echo "ERROR: [h100-node-setup] char class nvidia-caps-imex-channels absent from /proc/devices after retries; cannot create $CHANNEL" >&2 + exit 1 +fi +if mkdir -p /dev/nvidia-caps-imex-channels \ + && mknod "$CHANNEL" c "$MAJOR" 0 \ + && chmod 0666 "$CHANNEL"; then + echo "[h100-node-setup] created IMEX $CHANNEL (char major $MAJOR)" +else + echo "ERROR: [h100-node-setup] failed to create IMEX $CHANNEL (char major $MAJOR)" >&2 + exit 1 +fi +EOFHELPER +chmod 0755 "$IMEX_HELPER" + +# Create the channel for the current boot now. Run under `if` so a creation +# failure logs but never trips `set -e`: a node without fabric handles is +# recoverable, a failed cloud-init is not. +if ! "$IMEX_HELPER"; then + echo "ERROR: [h100-node-setup] IMEX channel not created for the current boot (see above); continuing" >&2 +fi + +cat >/etc/systemd/system/nvidia-imex-channel.service <