Skip to content

Commit 349e97f

Browse files
committed
docs(adr): add ADR-0001 (Rust/kube-rs) and ADR-0002 (cold-start as state)
ADR-0001 records why the operator is built in Rust with kube-rs rather than Go with controller-runtime: the operator model matters more than the language, continuity with the project's Rust profiling tools, and kube-rs being mature enough for the job — while honestly accepting the costs (Go's more mature ecosystem and wider hiring overlap). ADR-0002 records the decision that distinguishes this operator from a generic Deployment wrapper: modeling cold start as an explicit Pending -> Warming -> Ready lifecycle state, justified by the probe's measurement that "running" and "able to serve" are different moments for an LLM server. It also documents the deliberate placeholder data plane as a scope boundary.
1 parent 5798720 commit 349e97f

2 files changed

Lines changed: 80 additions & 0 deletions

File tree

docs/adr/0001-rust-and-kube-rs.md

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
# ADR-0001: Build the operator in Rust with kube-rs
2+
3+
- Status: Accepted
4+
- Date: 2026-05-31
5+
6+
## Context
7+
8+
This operator needs a control plane: a CRD, a reconcile loop, owned-resource management, and a status machine. The dominant ecosystem for building Kubernetes operators is Go — Kubernetes itself, controller-runtime, kubebuilder, and operator-sdk are all Go, and the overwhelming majority of production operators are written in it. Choosing the implementation language is therefore a real decision, not a default, and it deserves a record.
9+
10+
The realistic options were:
11+
12+
1. **Go with controller-runtime / kubebuilder** — the ecosystem standard.
13+
2. **Rust with kube-rs** — a mature but smaller operator framework.
14+
15+
## Decision
16+
17+
Build the operator in Rust with kube-rs.
18+
19+
## Rationale
20+
21+
The decision rests on three points, in order of weight.
22+
23+
1. **The operator model matters more than the language.** What a reviewer or employer should take from this project is that the author understands the Kubernetes control-plane model: custom resources, the reconcile loop, server-side apply, owner references and garbage collection, and the status subresource. That understanding is language-agnostic. Demonstrating it in Rust — where the type system makes the resource shapes and the reconcile contract explicit — is if anything a stronger demonstration that the model is understood rather than scaffolded.
24+
25+
2. **Continuity with the project's existing tooling.** This operator is the operational half of a cold-start research line whose measurement tools (vllm-coldstart-probe, inferscope) are already in Rust. Keeping the operator in the same language keeps the toolchain, the idioms, and the maintainer's productivity coherent across the whole line, rather than splitting it across two ecosystems for one component.
26+
27+
3. **kube-rs is mature enough not to be an obstacle.** kube-rs 2.x provides the CustomResource derive, a Controller runtime with watches and owned-resource retriggering, server-side apply, and status subresource support — everything this operator needs. The reconcile loop, ownership GC, and status machine were all implemented and verified end-to-end against a live cluster without hitting framework limitations.
28+
29+
## Consequences
30+
31+
### What we accept by not using Go
32+
33+
This is the honest cost of the decision, stated plainly:
34+
35+
- **A smaller ecosystem.** Go's operator tooling is more mature: kubebuilder scaffolds projects and RBAC, controller-runtime has a larger body of examples, envtest provides a standard integration-testing harness. In Rust these are either lighter or assembled by hand (here, an end-to-end test on an ephemeral kind cluster in CI plays the role envtest would in Go).
36+
- **A smaller hiring overlap.** Most teams operating Kubernetes write their controllers in Go. A Rust operator demonstrates the model but is not in the language those teams maintain day to day.
37+
38+
### Why we accept it
39+
40+
The project's purpose is to demonstrate that cold start can be made a first-class control-plane signal, and to demonstrate control-plane competence. Both goals are served by a correct, tested, well-documented operator in the language the author is most effective in. The cost — a less mature framework and a narrower language overlap — is outweighed, for a portfolio control plane built and maintained by one person, by correctness, coherence, and clarity.
41+
42+
A reviewer who maintains operators in Go should read this project as evidence that the author understands the operator model and can pick up controller-runtime quickly, not as a claim that Rust is the better choice for their stack.
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# ADR-0002: Model cold start as a first-class lifecycle state
2+
3+
- Status: Accepted
4+
- Date: 2026-05-31
5+
6+
## Context
7+
8+
Kubernetes already has a readiness model: a pod becomes Ready when its readiness probe passes. For most workloads that is enough. For an LLM inference server it is not, because there is a gap between two events that Kubernetes collapses into one:
9+
10+
- the process is up and accepting connections, and
11+
- the model is loaded, the GPU is warm, and the server can actually serve a token within normal latency.
12+
13+
The vllm-coldstart-probe study measured this gap directly: for a 7B model the cold start is dominated not by disk I/O but by GPU warmup and synchronization after the process is already running, and enabling CUDA graphs makes the gap several times larger. A generic operator that wraps a Deployment inherits Kubernetes' single notion of readiness and therefore cannot distinguish "running" from "warm". An operator born from the profiling work can.
14+
15+
## Decision
16+
17+
Make cold start an explicit, observable lifecycle state on the custom resource, rather than leaving it implicit in pod readiness.
18+
19+
The VllmService status exposes a phase derived from the owned Deployment's ready replicas:
20+
21+
- **Pending** — Deployment created, no replica ready.
22+
- **Warming** — at least one replica exists but not all desired replicas are ready: the process is up, the server is warming.
23+
- **Ready** — all desired replicas are ready: warm and able to serve, not merely running.
24+
25+
The warmupStrategy field (Eager / Graph) is part of the same decision: it lets the operator configure the cold-start/throughput trade-off the profiling work quantified, instead of treating it as an opaque deployment detail.
26+
27+
## Rationale
28+
29+
This is the decision that makes the operator more than a Deployment wrapper, and it is not clonable from a generic template: it exists because the cold-start cost was measured, and the measurement is what justifies treating warmth as a state worth modeling. The phase is derived from real Deployment status, not a timer, so it reflects the cluster's actual condition.
30+
31+
## Consequences
32+
33+
- The status is honest about the difference between "running" and "able to serve", which is the difference that matters for scale-to-zero and autoscaling decisions on inference workloads.
34+
- Future work can attach the measured cost to the state — for example, surfacing the expected cold-start cost per warmupStrategy in the status, so an operator can choose a strategy on the basis of the profiling data rather than a guess.
35+
36+
## Note on the placeholder data plane
37+
38+
On a CPU-only kind cluster the managed pod is a documented placeholder image, and warmupStrategy is recorded as an environment variable rather than driving a real vLLM flag. This is a deliberate scope boundary: the control plane (the lifecycle, ownership, and status modeling described here) is real and tested; the data plane is stubbed because a GPU and a real vLLM image are out of scope for a reproducible local/CI cluster. Validating that the Warming to Ready transition tracks the measured cold start on real vLLM is the next step, recorded in the README roadmap.

0 commit comments

Comments
 (0)