|
| 1 | +# GPU deployment status |
| 2 | + |
| 3 | +This operator's control plane is validated; running it against a real GPU |
| 4 | +cluster is documented here honestly, including what works and the open issue. |
| 5 | + |
| 6 | +## What is validated |
| 7 | + |
| 8 | +The control plane is fully exercised end-to-end on a `kind` cluster in CI: |
| 9 | +reconcile loop, the Pending -> Warming -> Ready state machine, owned-resource |
| 10 | +creation, owner references, and garbage collection. The GPU-specific code |
| 11 | +paths (image, `nvidia.com/gpu` limit, `runtimeClassName: nvidia`, and the |
| 12 | +readiness probe gating) are unit-tested and confirmed to render correctly, |
| 13 | +with `gpu=0` leaving them absent so CI stays GPU-free. |
| 14 | + |
| 15 | +The cold-start cost the Warming->Ready transition observes is measured |
| 16 | +separately on real A10 hardware; see BENCHMARKS.md. |
| 17 | + |
| 18 | +## Open issue: GPU scheduling on single-node K3s |
| 19 | + |
| 20 | +Bringing up GPU scheduling on a single-node K3s cluster (Lambda Cloud A10) |
| 21 | +hit a reproducible problem at the cluster layer, not in the operator: |
| 22 | + |
| 23 | +- K3s 1.35 does not auto-detect the NVIDIA container runtime even with the |
| 24 | + toolkit present; a hand-written `config.toml.tmpl` either failed to |
| 25 | + register the runtime or broke K3s startup (version-sensitive template). |
| 26 | +- The NVIDIA GPU Operator (Helm) installs cleanly and the CUDA validator |
| 27 | + confirms the GPU is usable, but with `CONTAINERD_SET_AS_DEFAULT=true` the |
| 28 | + containerd restart leaves K3s's flannel CNI uninitialized: the node goes |
| 29 | + `NotReady` and pods (including the operator's own validator) cannot schedule. |
| 30 | + |
| 31 | +The GPU itself works; the failure is the interaction between the GPU |
| 32 | +Operator's containerd reconfiguration and K3s's embedded CNI on a single node. |
| 33 | +This is a known sharp edge of K3s + GPU, not a defect in this operator. |
| 34 | + |
| 35 | +## Next step |
| 36 | + |
| 37 | +The robust path is a managed or kubeadm cluster, where the GPU Operator is |
| 38 | +heavily tested and does not contend with K3s's embedded CNI. The operator |
| 39 | +code is ready for it: it already emits `runtimeClassName: nvidia` and the GPU |
| 40 | +resource limit. Validating the full Warming->Ready transition against vLLM on |
| 41 | +such a cluster is the remaining work. |
0 commit comments