Skip to content

Commit fe34133

Browse files
committed
docs: document GPU deployment status and the K3s+CNI open issue
Honestly record what is validated (control plane on kind, GPU code paths unit-tested) versus the open issue: on single-node K3s the GPU Operator's containerd reconfiguration leaves flannel's CNI uninitialized when nvidia is set as default runtime, so the node goes NotReady. The GPU works (CUDA validator passes); the failure is a known K3s+GPU sharp edge, not an operator defect. Next step is a managed/kubeadm cluster where the operator code (which already emits runtimeClassName: nvidia) can be validated end-to-end.
1 parent 6c5b587 commit fe34133

1 file changed

Lines changed: 41 additions & 0 deletions

File tree

docs/GPU-DEPLOYMENT.md

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
# GPU deployment status
2+
3+
This operator's control plane is validated; running it against a real GPU
4+
cluster is documented here honestly, including what works and the open issue.
5+
6+
## What is validated
7+
8+
The control plane is fully exercised end-to-end on a `kind` cluster in CI:
9+
reconcile loop, the Pending -> Warming -> Ready state machine, owned-resource
10+
creation, owner references, and garbage collection. The GPU-specific code
11+
paths (image, `nvidia.com/gpu` limit, `runtimeClassName: nvidia`, and the
12+
readiness probe gating) are unit-tested and confirmed to render correctly,
13+
with `gpu=0` leaving them absent so CI stays GPU-free.
14+
15+
The cold-start cost the Warming->Ready transition observes is measured
16+
separately on real A10 hardware; see BENCHMARKS.md.
17+
18+
## Open issue: GPU scheduling on single-node K3s
19+
20+
Bringing up GPU scheduling on a single-node K3s cluster (Lambda Cloud A10)
21+
hit a reproducible problem at the cluster layer, not in the operator:
22+
23+
- K3s 1.35 does not auto-detect the NVIDIA container runtime even with the
24+
toolkit present; a hand-written `config.toml.tmpl` either failed to
25+
register the runtime or broke K3s startup (version-sensitive template).
26+
- The NVIDIA GPU Operator (Helm) installs cleanly and the CUDA validator
27+
confirms the GPU is usable, but with `CONTAINERD_SET_AS_DEFAULT=true` the
28+
containerd restart leaves K3s's flannel CNI uninitialized: the node goes
29+
`NotReady` and pods (including the operator's own validator) cannot schedule.
30+
31+
The GPU itself works; the failure is the interaction between the GPU
32+
Operator's containerd reconfiguration and K3s's embedded CNI on a single node.
33+
This is a known sharp edge of K3s + GPU, not a defect in this operator.
34+
35+
## Next step
36+
37+
The robust path is a managed or kubeadm cluster, where the GPU Operator is
38+
heavily tested and does not contend with K3s's embedded CNI. The operator
39+
code is ready for it: it already emits `runtimeClassName: nvidia` and the GPU
40+
resource limit. Validating the full Warming->Ready transition against vLLM on
41+
such a cluster is the remaining work.

0 commit comments

Comments
 (0)