Skip to content

Commit ccd3d09

Browse files
scotwellsclaude
scotwells
and
claude
committed
docs(quota): edge + single-tenant local quota enforcement RFC (#171)
Brings the consolidated RFC onto the branch: it covers the two non-management-plane deployments (edge/remote Milo and flat single-tenant local) and corrects the assumption that quota enforcement implies mode==milo. Also notes in the single-tenant sample config that Instance names must be unique across cell namespaces, since flat ResourceClaims are keyed by instance-<name> in one fixed project namespace (cross-namespace collision deferred, not fixed here). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 563f950 commit ccd3d09

2 files changed

Lines changed: 293 additions & 0 deletions

File tree

‎config/samples/single-tenant-quota-config.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,9 @@ data:
3434
# local quota is active and this is unset.
3535
localQuotaProjectID: my-project
3636
# Namespace where flat ResourceClaims are created. Defaults to "default".
37+
# Because every claim lands here keyed by instance-<name>, Instance names
38+
# must be unique across all cell namespaces: two same-named Instances in
39+
# different namespaces would share one claim and under-count quota.
3740
localQuotaProjectNamespace: default
3841
featureFlags:
3942
# Leave off: local quota supersedes the edge forwarder (the direct watch

‎docs/compute/development/rfcs/quota-edge-integration.md‎

Lines changed: 290 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,290 @@
1+
---
2+
status: mixed — management-plane quota (shipped) + edge & single-tenant (proposed)
3+
---
4+
5+
# Quota Edge Integration: Routing ResourceClaims to Project Control Planes
6+
7+
> The management-plane claim-routing design is **implemented and shipped** (last
8+
> verified 2026-05-26). The edge-cell and flat single-tenant designs below are
9+
> **proposed, not yet implemented**. Fixes
10+
> [#171](https://github.com/datum-cloud/compute/issues/171).
11+
12+
## Table of Contents
13+
14+
- [Summary](#summary)
15+
- [Problem Statement](#problem-statement)
16+
- [Three deployment topologies](#three-deployment-topologies)
17+
- [Design](#design)
18+
- [Watch enablement follows where claims live](#watch-enablement-follows-where-claims-live)
19+
- [Claim CRUD routes to the owning ledger](#claim-crud-routes-to-the-owning-ledger)
20+
- [Edge cell: the forwarder pattern](#edge-cell-the-forwarder-pattern)
21+
- [Flat single-tenant: co-located quota and fixed-project identity](#flat-single-tenant-co-located-quota-and-fixed-project-identity)
22+
- [Scoping the watch cache](#scoping-the-watch-cache)
23+
- [Alternatives considered](#alternatives-considered)
24+
- [RBAC dependency and failure modes](#rbac-dependency-and-failure-modes)
25+
- [Known limitation: cell-wide unique Instance names](#known-limitation-cell-wide-unique-instance-names)
26+
- [Rollout](#rollout)
27+
- [Risks](#risks)
28+
- [Open questions](#open-questions)
29+
- [Out of Scope](#out-of-scope)
30+
31+
---
32+
33+
## Summary
34+
35+
Datum operators want to enforce quota on the Instances their customers run — and not
36+
only in the central management plane, but wherever compute runs: on **edge cells**
37+
close to users and on **flat single-tenant clusters**. Today quota enforcement works
38+
only in the central management plane. Turn it on anywhere else and one of two bad
39+
things happens: quota silently never evaluates (claims are recorded where the quota
40+
system never reads them, so every Instance runs ungated), or compute-manager crashes
41+
outright ([#171](https://github.com/datum-cloud/compute/issues/171)).
42+
43+
This RFC makes quota enforcement work across all three deployments, so an operator can
44+
stand up a quota-enforced edge cell or single-tenant cluster and have customer
45+
Instances gated on real quota grants. The shipped half routes each Instance's quota
46+
claim to the control plane that owns the project's quota ledger. The proposed half
47+
extends the same idea to deployments where that ledger is either remote (edge cells)
48+
or co-located on the one cluster (flat single-tenant).
49+
50+
---
51+
52+
## Problem Statement
53+
54+
**What the operator saw.** Enabling quota outside the management plane did nothing
55+
useful, or worse. In the best case every Instance sat waiting on a quota grant that
56+
never came, with no error and no signal — because the claim recording the Instance's
57+
demand was written somewhere the quota system never reads. In the worst case (edge
58+
cells) compute-manager failed to start at all: it tried to watch for quota claims on a
59+
cluster that doesn't host the quota API, and crashed on startup.
60+
61+
Both failures share one root: the system assumed quota claims live on, and are watched
62+
from, whatever cluster the controller happens to run on. That assumption holds only in
63+
the management plane. The design below replaces it with an explicit model of where each
64+
deployment's quota ledger actually lives.
65+
66+
---
67+
68+
## Three deployment topologies
69+
70+
Quota enforcement spans three deployments, each serving a different customer shape and
71+
keeping its quota ledger in a different place:
72+
73+
| Topology | Serves | Where the quota ledger / ResourceClaims live | Claim watch |
74+
|---|---|---|---|
75+
| **Management plane** | Datum's central control plane, across many customer projects | Per-project Milo control planes | on |
76+
| **Edge cell** | Compute close to users; ledger back in a remote Milo | Remote Milo project control planes (not on the cell) | off by default; on via the forwarder pattern |
77+
| **Flat single-tenant** | A single customer's own cluster, no central Milo in the path | Co-located on the one cluster | on |
78+
79+
The management-plane topology is shipped; the other two are the proposed additions.
80+
The differences that matter architecturally are *where the claims live* and, following
81+
from that, *whether the controller can watch them directly*.
82+
83+
---
84+
85+
## Design
86+
87+
### Watch enablement follows where claims live
88+
89+
The core decision: **enable the quota-claim watch based on where the claims actually
90+
live — whether the cluster being watched serves the quota API — not on the deployment's
91+
cluster-discovery mode.**
92+
93+
Keying the watch on discovery mode conflated two independent things. A cell running as
94+
a single-cluster deployment may keep its ledger remote (edge) or local (flat
95+
single-tenant); the watch should be off in the first case and on in the second. Keying
96+
instead on whether the target cluster serves the quota API gets both right — and
97+
because the very check that tells the watch the claim kind exists is the one that
98+
decides the gate, the gate can never disagree with the watch, so it cannot re-introduce
99+
the startup crash:
100+
101+
> watch quota claims ⟺ the watched cluster serves the quota API
102+
103+
A transient inability to determine this must **fail loud** (refuse to start) rather
104+
than silently resolve to "off," which would masquerade as a healthy deployment while
105+
dropping every grant event.
106+
107+
### Claim CRUD routes to the owning ledger
108+
109+
Creating and deleting claims must target the same ledger the watch reads. In the
110+
management-plane and edge topologies the owning ledger is the project's control plane,
111+
so claim writes are directed there; on a flat single-tenant cluster there is no remote
112+
control plane, so writes go to the local cluster.
113+
114+
This is not cosmetic. If writes and the watch target different ledgers, claims are
115+
created in one place and observed in another, so grants are never seen even though
116+
everything appears wired. The invariant is simply that an Instance's claim is written
117+
to, and read from, the **same** ledger.
118+
119+
### Edge cell: the forwarder pattern
120+
121+
On an edge cell the quota ledger lives in a remote Milo, so the Instance controller —
122+
which runs against the local cell — cannot watch it directly. The design keeps the main
123+
controllers untouched on the local cell and adds a **separate quota watcher**: a
124+
distinct provider that engages the remote project control planes, watches quota claims
125+
there, and forwards grant/denial events back to the local Instance controller, which
126+
then clears the Instance's scheduling gate.
127+
128+
```
129+
Edge cell — compute-manager
130+
┌──────────────────────────────────────────────────────────────────┐
131+
│ │
132+
│ Instance controller Quota watcher │
133+
│ (runs on the local cell) (separate provider, │
134+
│ ┌────────────────────────┐ default-off) │
135+
│ │ creates / deletes │ ┌───────────────────────┐ │
136+
│ │ claims in the remote │ │ watches claims in the │ │
137+
│ │ ledger; gates the │◀─grant─┤ remote ledger; │ │
138+
│ │ Instance on the grant │ events│ forwards grant events │ │
139+
│ └───────────┬────────────┘ └───────────┬───────────┘ │
140+
└───────────────┼───────────────────────────────────┼────────────────┘
141+
│ create / delete │ read (list/watch)
142+
▼ ▼
143+
Remote Milo project control planes = the quota ledger
144+
```
145+
146+
This watcher is gated behind a **default-off capability flag**. With the flag off, the
147+
edge cell registers no remote watch at all — so it cannot crash — and falls back to a
148+
periodic backoff re-check of each pending Instance's claim. That fallback is slower to
149+
observe a grant but always correct; it is the safe default, and the flag exists to
150+
remove the latency once a deployment is ready. The watcher runs on every replica, but
151+
only the leader acts on forwarded events; a non-leader harmlessly drops them.
152+
153+
### Flat single-tenant: co-located quota and fixed-project identity
154+
155+
A flat single-tenant deployment co-locates the quota API and compute on one cluster, so
156+
the existing local watch already engages the cluster that holds the claims — no
157+
forwarder, no flag, no remote provider. It is reached simply by letting the watch run
158+
when the quota API is present locally (the decision above). The two proposed paths are
159+
mutually exclusive by construction: if the quota API is local, the edge forwarder is
160+
never engaged.
161+
162+
The one genuinely new concept is **identity**. In the management-plane and edge
163+
topologies the project that owns a claim is derived from platform namespace-ownership
164+
labels. A flat single-tenant deployment may not run that platform labeling layer, so it
165+
instead names its single project explicitly through configuration, and every claim is
166+
attributed to that project. If local quota is enabled without a configured project, the
167+
deployment refuses to start rather than emit claims the evaluator cannot attribute.
168+
169+
### Scoping the watch cache
170+
171+
A quota watcher that engages many project control planes must **scope what it caches to
172+
only the claims this deployment created**. Watching a kind without scoping the cache
173+
pulls every claim across every engaged project into memory — a known cause of
174+
out-of-memory failures on this platform. Scoping is a property of the cache itself, not
175+
a post-hoc filter on events: filtering events after the fact still caches everything.
176+
This bounds memory per connection, though the number of watch connections still grows
177+
with the number of engaged projects (see [Risks](#risks)).
178+
179+
---
180+
181+
## Alternatives considered
182+
183+
- **One unified multi-cluster provider spanning the cell and the remote ledger.**
184+
Rejected: folding the remote ledger into the cell's own provider destabilizes the
185+
cell's control path — it breaks the single-cluster assumption the cell's controllers
186+
rely on to run at all. A separate watcher keeps the cell path untouched.
187+
- **Install the quota API locally on edge cells.** Rejected: an edge cell should not
188+
host the authoritative quota ledger. The ledger lives back in the platform; the edge
189+
reads it, it doesn't own it.
190+
- **Rely only on the periodic requeue, with no watch.** Kept — but as the fallback
191+
floor, not the whole answer. It is always correct and is the safe default, but grants
192+
lag until the next re-check, so it can't be the sole mechanism where timely
193+
enforcement matters.
194+
195+
---
196+
197+
## RBAC dependency and failure modes
198+
199+
The edge forwarder reads a ledger it does not own, so it depends on permissions granted
200+
in each remote project control plane — read access to the quota claims, and the ability
201+
to discover which projects exist. Neither can be granted from this repository; both are
202+
platform-side prerequisites.
203+
204+
The two grants **fail differently**, and an operator must know which is which:
205+
206+
- **Project discovery missing → fail loud.** Without the ability to enumerate projects
207+
the watcher cannot start; it crashes visibly (the same failure class as #171,
208+
relocated to the watcher). Immediate and hard to miss.
209+
- **Claim read access missing → silent degrade.** The watcher starts, but per-project
210+
engagement fails quietly and no grant events arrive. Enforcement falls back to the
211+
periodic backoff re-check. It presents as "quota works but grants are slow" — the
212+
dangerous case to watch for.
213+
214+
Because of this asymmetry, both grants must be verified explicitly before enabling the
215+
watcher on a given deployment.
216+
217+
---
218+
219+
## Known limitation: cell-wide unique Instance names
220+
221+
**Deferred — an operator constraint, not a rollout blocker.** In a flat single-tenant
222+
deployment all of a project's quota claims share one space, and a claim's identity is
223+
derived from its Instance's name. Two Instances with the **same name in different
224+
namespaces** therefore collide on a single claim: one can overwrite or block the other,
225+
so quota is attributed to the wrong Instance or a legitimate Instance is wrongly gated.
226+
This cannot happen in the management-plane or edge topologies, where claims are
227+
separated per project control plane and per project namespace.
228+
229+
Until claim identity is qualified by namespace, the operator rule for flat
230+
single-tenant mode is: **Instance names must be unique across the whole cell**, not
231+
merely within a namespace. Qualifying claim identity by namespace is the fix, tracked
232+
separately.
233+
234+
---
235+
236+
## Rollout
237+
238+
The proposed capabilities ship dark and enable per deployment:
239+
240+
1. **Ship with the edge watcher off.** Every deployment is safe immediately — edge
241+
cells no longer crash, and enforcement uses the backoff fallback.
242+
2. **Confirm the platform-side RBAC** for a deployment before enabling its watcher —
243+
both grants (see [RBAC dependency and failure modes](#rbac-dependency-and-failure-modes)).
244+
3. **Enable per deployment**, watching memory as the watcher engages projects. The off
245+
state remains a safe fallback if a deployment misbehaves.
246+
247+
Flat single-tenant needs no flag: it activates when the quota API is present locally and
248+
a project is configured.
249+
250+
---
251+
252+
## Risks
253+
254+
- **Unbounded watch cache.** If cache scoping regresses to an after-the-fact event
255+
filter, the watcher caches every claim across every project and can run out of memory.
256+
Scoping the cache (not the events) is the mitigation.
257+
- **Watch-connection fan-out.** Cache scoping bounds memory *per connection*, but the
258+
watcher still opens one connection per engaged project, so connection count grows with
259+
the number of projects — on a shared control plane, with the number of edge
260+
deployments times projects. Restricting engagement to only the projects a deployment
261+
actually serves is the future mitigation.
262+
- **Silent RBAC degrade.** A missing claim-read grant degrades quietly to the slower
263+
fallback rather than failing loudly; operators can mistake it for healthy enforcement.
264+
- **Fallback latency.** With the edge watcher off, a grant is observed only on the
265+
backoff re-check, not immediately. Accepted as the safe default.
266+
267+
---
268+
269+
## Open questions
270+
271+
1. Can the credential a flat/edge deployment already carries enumerate the platform's
272+
projects for discovery, or is a separate discovery path needed? (Blocks enabling the
273+
edge watcher.)
274+
2. Should a watcher engage only the projects a deployment actually serves, rather than
275+
every project it can see? This bounds connection fan-out directly.
276+
3. Does the co-located quota evaluator actually grant claims attributed to the configured
277+
flat-tenant project (is that project entitled), or do they stall? This is platform-side
278+
behavior, verifiable only in a running deployment.
279+
280+
---
281+
282+
## Out of Scope
283+
284+
**WorkloadDeployment quota.** This design covers Instance-level quota claims only.
285+
Whether WorkloadDeployments should also emit claims, and at what resources, is a
286+
separate design.
287+
288+
**Migration of mis-written claims.** Claims recorded on the wrong cluster before the
289+
shipped fix remain there; they are never evaluated and are not cleaned up by the
290+
corrected paths. Removing them is an operational task, not part of this design.

0 commit comments

Comments
 (0)