docs(quota): edge + single-tenant local quota enforcement RFC (#171)

Adds the RFC covering the two non-management-plane deployments — edge/remote Milo
and flat single-tenant local — and correcting the assumption that quota
enforcement implies mode==milo. Edge cells enforce quota via the always-on
backoff/requeue safety net; a live edge claim watch is a deferred future option.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: scotwells

docs/compute/development/rfcs/quota-edge-integration.md

@@ -0,0 +1,254 @@
+---
+status: mixed — management-plane quota (shipped) + edge (backoff) & single-tenant (proposed)
+---
+
+# Quota Edge Integration: Routing ResourceClaims to Project Control Planes
+
+> The management-plane claim-routing design is **implemented and shipped** (last
+> verified 2026-05-26). The edge-cell and flat single-tenant designs below are
+> **proposed, not yet implemented**. Fixes
+> [#171](https://github.com/datum-cloud/compute/issues/171).
+
+## Table of Contents
+
+- [Summary](#summary)
+- [Problem Statement](#problem-statement)
+- [Three deployment topologies](#three-deployment-topologies)
+- [Design](#design)
+  - [The live watch runs where claims are co-located](#the-live-watch-runs-where-claims-are-co-located)
+  - [Claim CRUD routes to the owning ledger](#claim-crud-routes-to-the-owning-ledger)
+  - [The backoff safety net](#the-backoff-safety-net)
+  - [Flat single-tenant: co-located quota and fixed-project identity](#flat-single-tenant-co-located-quota-and-fixed-project-identity)
+- [Alternatives considered](#alternatives-considered)
+- [RBAC dependency](#rbac-dependency)
+- [Known limitation: cell-wide unique Instance names](#known-limitation-cell-wide-unique-instance-names)
+- [Rollout](#rollout)
+- [Risks](#risks)
+- [Open questions](#open-questions)
+- [Out of Scope](#out-of-scope)
+
+---
+
+## Summary
+
+Datum operators want to enforce quota on the Instances their customers run — and not
+only in the central management plane, but wherever compute runs: on **edge cells**
+close to users and on **flat single-tenant clusters**. Today quota enforcement works
+only in the central management plane. Turn it on anywhere else and one of two bad
+things happens: quota silently never evaluates (claims are recorded where the quota
+system never reads them, so every Instance runs ungated), or compute-manager crashes
+outright ([#171](https://github.com/datum-cloud/compute/issues/171)).
+
+This RFC makes quota enforcement work across all three deployments, so an operator can
+stand up a quota-enforced edge cell or single-tenant cluster and have customer
+Instances gated on real quota grants. The shipped half routes each Instance's quota
+claim to the control plane that owns the project's quota ledger and watches those claims
+where the manager already engages them. The proposed half extends enforcement to two
+more deployments: **flat single-tenant** clusters, where the ledger is co-located and
+watched directly; and **edge cells**, where the ledger is remote and enforcement rides
+the always-on backoff reconcile rather than a live watch.
+
+---
+
+## Problem Statement
+
+**What the operator saw.** Enabling quota outside the management plane did nothing
+useful, or worse. In the best case every Instance sat waiting on a quota grant that
+never came, with no error and no signal — because the claim recording the Instance's
+demand was written somewhere the quota system never reads. In the worst case (edge
+cells) compute-manager failed to start at all: it tried to watch for quota claims on a
+cluster that doesn't host the quota API, and crashed on startup.
+
+Both failures share one root: the system assumed quota claims live on, and are watched
+from, whatever cluster the controller happens to run on. That assumption holds only in
+the management plane. The design below replaces it with an explicit model of where each
+deployment's quota ledger actually lives, and of how each learns that a claim has been
+granted.
+
+---
+
+## Three deployment topologies
+
+Quota enforcement spans three deployments, each serving a different customer shape,
+keeping its quota ledger in a different place, and learning about grants differently:
+
+| Topology | Serves | Where the quota ledger / ResourceClaims live | How grants are observed |
+|---|---|---|---|
+| **Management plane** | Datum's central control plane, across many customer projects | Per-project Milo control planes, which the manager already engages | Live direct watch |
+| **Edge cell** | Compute close to users; ledger back in a remote Milo | Remote Milo, per-project control planes — **not** on the cell | **Periodic backoff reconcile — no live watch** |
+| **Flat single-tenant** | A single customer's own cluster, no central Milo in the path | Co-located on the one cluster | Live direct watch |
+
+The management-plane topology is shipped; the other two are the proposed additions. The
+distinction that matters architecturally is whether the claims sit on a cluster the
+manager already watches. Where they do (management plane, single-tenant) enforcement can
+observe grants live; where they don't (edge) it relies on the backoff floor.
+
+---
+
+## Design
+
+### The live watch runs where claims are co-located
+
+The core decision: **a live claim watch runs only on the clusters the manager already
+engages that serve the quota API** — the management plane's per-project provider
+clusters, and a flat single-tenant cluster's local API. We do **not** stand up
+additional watches against clusters the manager isn't already engaging.
+
+An edge cell's ledger fails that test twice over: its claims are remote, and they are
+split across one control plane per project the cell serves. So an edge cell registers no
+live claim watch — which is also exactly what stops the crash, since its local cluster
+does not serve the quota API and never should have been asked to watch that kind:
+
+> watch claims live ⟺ a cluster the manager already engages serves the quota API
+
+A transient inability to determine this must **fail loud** (refuse to start) rather than
+silently resolve to "off," which would masquerade as a healthy deployment.
+
+### Claim CRUD routes to the owning ledger
+
+Creating, reading, and deleting an Instance's claim targets the control plane that owns
+the project's quota ledger: the project control plane in the management-plane and edge
+topologies, the local cluster in flat single-tenant. Whatever observes a grant must read
+from the same ledger the claim was written to — if writes and reads target different
+ledgers, claims are created in one place and looked for in another, so grants are never
+seen even though everything appears wired.
+
+At the edge this routing is also how enforcement stays live without a watch: each
+backoff reconcile creates-or-reads the Instance's remote claim and checks whether it has
+been granted.
+
+### The backoff safety net
+
+Enforcement has always had an always-on floor: while an Instance's quota is not yet
+granted, its reconcile re-runs on a backing-off schedule and re-checks the claim each
+time. This floor is **universal** — every topology falls back to it — and at the **edge
+it is the sole liveness mechanism**. Grants there are observed within seconds and
+self-heal as the schedule backs off; there is simply no live watch to short-circuit the
+wait.
+
+The tradeoff is latency, not correctness. An Instance is never ungated until its quota
+is actually granted; the only cost of relying on backoff is how quickly a grant is
+*noticed*. That cost is acceptable at the edge, which is why the live edge watch is
+deferred (see [Alternatives considered](#alternatives-considered)).
+
+### Flat single-tenant: co-located quota and fixed-project identity
+
+A flat single-tenant deployment co-locates the quota API and compute on one cluster, so
+the manager already engages the cluster that holds the claims — the live watch runs
+there directly, with no remote hop. It is reached simply by letting the watch run when
+the quota API is present locally (the decision above).
+
+The one genuinely new concept is **identity**. In the management-plane and edge
+topologies the project that owns a claim is derived from platform namespace-ownership
+labels. A flat single-tenant deployment may not run that platform labeling layer, so it
+instead names its single project explicitly through configuration, and every claim is
+attributed to that project. If local quota is enabled without a configured project, the
+deployment refuses to start rather than emit claims the evaluator cannot attribute.
+
+---
+
+## Alternatives considered
+
+- **A live edge watch via a per-project-control-plane forwarder.** Considered and
+  **deferred**. Because an edge cell's claims are split across one control plane per
+  project it serves, a live edge watch means establishing and maintaining a watch
+  against *each* of those control planes — fan-out that grows with the projects a cell
+  serves. The always-on backoff already delivers grants within seconds, so the machinery
+  isn't worth it now. Revisit if edge grant latency ever becomes a product concern.
+- **A single unified multi-cluster provider spanning the cell and the remote ledger.**
+  Rejected: folding the remote ledger into the cell's own provider destabilizes the
+  cell's control path — it breaks the single-cluster assumption the cell's controllers
+  rely on to run at all.
+- **Install the quota API locally on edge cells.** Rejected: an edge cell should not
+  host the authoritative quota ledger. The ledger lives back in the platform; the edge
+  reads it, it doesn't own it.
+
+---
+
+## RBAC dependency
+
+Every topology that writes claims to a ledger it does not host depends on permissions
+there — but only for the operations it actually performs. None can be granted from this
+repository; all are platform-side prerequisites.
+
+- **Edge cell.** The cell creates, reads, and deletes claims in the remote per-project
+  control planes — it writes each claim and polls its grant on every backoff reconcile.
+  It needs those create/read/delete grants, but because it runs **no live watch** it
+  does **not** need list/watch there, and it needs no project-discovery permission.
+- **Flat single-tenant.** The manager needs access to the co-located quota API on its
+  own cluster.
+- **Management plane.** The pre-existing provider-cluster watch on project control
+  planes is unchanged (shipped).
+
+If an edge cell lacks the create/read access it needs, claim creation and grant-polling
+fail: the reconcile errors and requeues, and the Instance stays gated until access is
+fixed. That is a loud, self-retrying failure — not a silent bypass.
+
+---
+
+## Known limitation: cell-wide unique Instance names
+
+**Deferred — an operator constraint, not a rollout blocker.** In a flat single-tenant
+deployment all of a project's quota claims share one space, and a claim's identity is
+derived from its Instance's name. Two Instances with the **same name in different
+namespaces** therefore collide on a single claim: one can overwrite or block the other,
+so quota is attributed to the wrong Instance or a legitimate Instance is wrongly gated.
+This cannot happen in the management-plane or edge topologies, where claims are separated
+per project control plane and per project namespace.
+
+Until claim identity is qualified by namespace, the operator rule for flat single-tenant
+mode is: **Instance names must be unique across the whole cell**, not merely within a
+namespace. Qualifying claim identity by namespace is the fix, tracked separately.
+
+---
+
+## Rollout
+
+There is no capability flag to manage; the proposed changes take effect by deployment
+shape:
+
+1. **Edge cells stop crashing immediately.** The live watch is simply not registered
+   where the local cluster doesn't serve the quota API, and enforcement rides the
+   backoff floor. Confirm the cell's platform-side create/read access to the remote
+   ledger so that the claims it writes are actually evaluated.
+2. **Flat single-tenant** activates when the quota API is present locally and a project
+   is configured.
+3. **Management plane** is unchanged.
+
+---
+
+## Risks
+
+- **Edge grant latency.** With no live edge watch, a grant is observed on the backoff
+  schedule (seconds, backing off), not instantly. Accepted as the tradeoff of deferring
+  the edge watch; revisit if latency ever matters to the product.
+- **Edge access to the remote ledger.** If a cell lacks create/read on the remote
+  per-project claims, its Instances stay gated until access is fixed — correct (no
+  bypass), but it blocks scheduling and surfaces as recurring reconcile errors rather
+  than a single obvious failure.
+- **Flat single-tenant name collision.** Same-named Instances in different namespaces
+  contend for one claim; see [Known limitation](#known-limitation-cell-wide-unique-instance-names).
+
+---
+
+## Open questions
+
+1. Does the co-located quota evaluator actually grant claims attributed to the configured
+   flat-tenant project (is that project entitled), or do they stall? Platform-side
+   behavior, verifiable only in a running deployment.
+2. Is the per-project create/read access a cell needs actually in place across the
+   projects it serves? A rollout verification, not a design question.
+3. If edge grant latency ever matters, is a live per-project edge watch worth building?
+   (The revisit trigger for the deferred alternative above.)
+
+---
+
+## Out of Scope
+
+**WorkloadDeployment quota.** This design covers Instance-level quota claims only.
+Whether WorkloadDeployments should also emit claims, and at what resources, is a separate
+design.
+
+**Migration of mis-written claims.** Claims recorded on the wrong cluster before the
+shipped fix remain there; they are never evaluated and are not cleaned up by the
+corrected paths. Removing them is an operational task, not part of this design.