|
| 1 | +# OKE GPU Setup |
| 2 | + |
| 3 | +## GPU Stack Ownership |
| 4 | + |
| 5 | +OKE installs NVIDIA's device plugin automatically on every cluster, and |
| 6 | +Oracle's GPU node images preinstall the NVIDIA driver, container toolkit, and |
| 7 | +host MOFED. Which of those a GPU node pool actually has depends on how it was |
| 8 | +provisioned, and the AICR recipe must match it — the `gpuStack` configuration |
| 9 | +profile on the OKE family names the three qualified combinations: |
| 10 | + |
| 11 | +| Value | Driver / toolkit | `nvidia.com/gpu` advertiser | Pool shape | |
| 12 | +|---|---|---|---| |
| 13 | +| `oci-default` (default) | Oracle GPU node image | OKE's auto-installed device plugin | stock OKE with Oracle GPU images | |
| 14 | +| `operator-plugin` | Oracle GPU node image | GPU Operator's device plugin | OKE plugin disabled (node label or add-on removed) | |
| 15 | +| `operator-managed` | GPU Operator installs both | GPU Operator's device plugin | bring-your-own driverless image | |
| 16 | + |
| 17 | +MOFED is host-supplied in every value — Oracle's GPU images and the common |
| 18 | +bring-your-own images alike carry it, so `network-operator` never deploys |
| 19 | +`ofedDriver` on OKE, and the device plugin runs with `MOFED_ENABLED=false` |
| 20 | +(without it, k8s-device-plugin >= 0.19.0 with CDI floods every host RDMA |
| 21 | +uverb into every GPU pod and breaks NCCL). |
| 22 | + |
| 23 | +Select the mode at recipe generation; the paths it owns are locked at every |
| 24 | +output boundary: |
| 25 | + |
| 26 | +```shell |
| 27 | +# Stock OKE cluster (the default) — no flag needed |
| 28 | +aicr recipe --service oke --accelerator l40s --os ol --intent training -o recipe.yaml |
| 29 | + |
| 30 | +# OKE plugin disabled, image-supplied driver |
| 31 | +aicr recipe --service oke --accelerator l40s --os ol --intent training \ |
| 32 | + --profile gpuStack=operator-plugin -o recipe.yaml |
| 33 | + |
| 34 | +# Bring-your-own driverless image (e.g. a custom Ubuntu image) |
| 35 | +aicr recipe --service oke --accelerator gb200 --os ubuntu --intent training \ |
| 36 | + --profile gpuStack=operator-managed -o recipe.yaml |
| 37 | +``` |
| 38 | + |
| 39 | +## Default: Stock OKE (`oci-default`) |
| 40 | + |
| 41 | +A default-provisioned OKE cluster with Oracle GPU images needs zero setup: |
| 42 | +the image supplies the driver and toolkit, and OKE's device plugin advertises |
| 43 | +`nvidia.com/gpu`. The GPU Operator manages the rest of the stack with its own |
| 44 | +plugin disabled — running both plugins double-advertises the same GPUs, which |
| 45 | +the #1327 exactly-one-advertiser policy forbids. |
| 46 | + |
| 47 | +## Alternative: Let the GPU Operator's Plugin Advertise (`operator-plugin`) |
| 48 | + |
| 49 | +If you prefer the GPU Operator's device plugin (feature discovery, MIG, CDI |
| 50 | +control), disable OKE's plugin on the GPU pools and select the value: |
| 51 | + |
| 52 | +- **Per node pool (recommended, snapshot-visible):** add the node label |
| 53 | + `oci.oraclecloud.com/disable-gpu-device-plugin=true` to the pool's initial |
| 54 | + node labels at creation. |
| 55 | +- **Per cluster:** remove the `NvidiaGpuPlugin` cluster add-on |
| 56 | + (Terraform `addons = { NvidiaGpuPlugin = { remove = true } }`, or the |
| 57 | + add-on lifecycle API). Note this leaves no on-node marker. |
| 58 | + |
| 59 | +The driver still comes from the Oracle image — the GPU Operator installs |
| 60 | +nothing under this value. |
| 61 | + |
| 62 | +## Alternative: Bring-Your-Own Image (`operator-managed`) |
| 63 | + |
| 64 | +Custom images (OKE Ubuntu pools are always custom images) may ship no NVIDIA |
| 65 | +stack at all. Under `operator-managed` the GPU Operator installs the driver |
| 66 | +and toolkit, its device plugin advertises, and the DRA kubelet plugin reads |
| 67 | +the driver userspace from the operator install path |
| 68 | +(`/run/nvidia/driver` — the profile moves `nvidiaDriverRoot` in lockstep; |
| 69 | +see the driver-ownership coherence rules). Disable OKE's device plugin on |
| 70 | +these pools exactly as under `operator-plugin`. |
| 71 | + |
| 72 | +If your custom image DOES bake a driver (Oracle publishes downloadable |
| 73 | +Ubuntu GPU images with driver + CUDA + DOCA-OFED), use `operator-plugin` |
| 74 | +instead — a second, operator-installed driver on top of an image driver is |
| 75 | +an unqualified hybrid. |
| 76 | + |
| 77 | +## Validation |
| 78 | + |
| 79 | +The three values deploy differently, so `aicr validate` distinguishes them by |
| 80 | +**deployed ClusterPolicy state** (readiness pre-flight, fail closed): |
| 81 | +`K8s.policy.driver.enabled` and `K8s.policy.devicePlugin.enabled` must match |
| 82 | +the selected value. A recipe generated with the wrong mode for the cluster |
| 83 | +fails the pre-flight with remediation text before any check Jobs deploy. |
| 84 | +There is no generation-time gate yet: the snapshot cannot see a removed |
| 85 | +add-on, and pools disabled via the add-on (rather than the node label) leave |
| 86 | +no on-node marker. |
| 87 | + |
| 88 | +## Oracle Add-on Interactions |
| 89 | + |
| 90 | +Do **not** enable Oracle's `NvidiaGpuOperator` or `NvidiaNetworkOperator` |
| 91 | +managed add-ons alongside AICR bundles — the bundle deploys both operators |
| 92 | +itself, and two lifecycle managers fight over the same releases. The only |
| 93 | +Oracle GPU add-on compatible with AICR bundles is the device plugin |
| 94 | +(`NvidiaGpuPlugin`), and only under `oci-default`. |
| 95 | + |
| 96 | +## References |
| 97 | + |
| 98 | +- [OKE: Running GPU Workloads](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengrunninggpunodes.htm) |
| 99 | +- [OKE cluster add-ons](https://docs.oracle.com/en-us/iaas/Content/ContEng/Tasks/contengintroducingclusteraddons.htm) |
| 100 | +- [oci-hpc-oke quickstart](https://github.com/oracle-quickstart/oci-hpc-oke) — worker images, RDMA manifests |
| 101 | +- [AKS GPU Setup](aks-gpu-setup.md), [GKE GPU Setup](gke-gpu-setup.md) — the sibling `gpuStack` families |
0 commit comments