docs(quota): quota enforcement enhancement (#171)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: scotwells

docs/compute/development/enhancements/quota-enforcement/README.md

@@ -0,0 +1,253 @@
+---
+status: provisional
+stage: alpha
+latest-milestone: "v0.x"
+---
+
+<!--
+TODO (datum-cloud/enhancements process — not yet done):
+- Open a tracking issue in datum-cloud/enhancements and link it here. The doc's
+  final location (area directory + short title) is decided in that issue.
+- If this lands in the enhancements repo, place it under the compute area, e.g.
+  `enhancements/compute/quota-enforcement/README.md`.
+- Assign the PR to sponsoring approvers before moving status -> implementable.
+
+Status note: management-plane enforcement is already IMPLEMENTED; the edge and
+flat single-tenant designs are PROPOSED. Overall status is kept `provisional`
+until the proposed portion is approved.
+-->
+
+# Quota Enforcement Across Compute Deployments
+
+- [Summary](#summary)
+- [Motivation](#motivation)
+  - [Goals](#goals)
+  - [Non-Goals](#non-goals)
+- [Proposal](#proposal)
+  - [User Stories](#user-stories)
+  - [Notes/Constraints/Caveats](#notesconstraintscaveats)
+  - [Risks and Mitigations](#risks-and-mitigations)
+- [Design Details](#design-details)
+  - [Where claims live decides how grants are observed](#where-claims-live-decides-how-grants-are-observed)
+  - [Claims route to the owning ledger](#claims-route-to-the-owning-ledger)
+  - [Flat single-tenant identity](#flat-single-tenant-identity)
+  - [Access prerequisites](#access-prerequisites)
+- [Production Readiness Review Questionnaire](#production-readiness-review-questionnaire)
+  - [Feature Enablement and Rollback](#feature-enablement-and-rollback)
+  - [Monitoring Requirements](#monitoring-requirements)
+  - [Dependencies](#dependencies)
+  - [Scalability](#scalability)
+  - [Troubleshooting](#troubleshooting)
+- [Implementation History](#implementation-history)
+- [Drawbacks](#drawbacks)
+- [Alternatives](#alternatives)
+- [Infrastructure Needed](#infrastructure-needed)
+
+## Summary
+
+Customers should be able to run quota-enforced Instances on **any** Datum deployment —
+the central management plane, an edge cell, or a flat single-tenant cluster. Today they
+can't: enabling quota anywhere but the management plane either silently never evaluates
+(claims recorded where the quota system never reads them) or crashes compute-manager
+outright. This enhancement makes enforcement work in all three deployments, so an
+Instance is gated on a real quota grant wherever it runs. The single organizing idea is
+that a quota claim is created in — and read from — the control plane that owns the
+project's quota ledger, and how a grant is observed follows from where that ledger sits.
+
+## Motivation
+
+Quota is how the platform bounds what customers can consume. If it only works in the
+central management plane, then every deployment closer to the customer — edge cells and
+single-tenant clusters — either runs ungated or can't run compute-manager at all. The
+crash on edge cells is a hard blocker to standing compute up there. Making enforcement
+portable across deployments is what lets quota-bearing products ship beyond the
+management plane.
+
+### Goals
+
+- Enforce quota on Instances in all three deployments: management plane, edge cell, flat
+  single-tenant.
+- compute-manager starts and stays healthy with quota enabled in every deployment (no
+  startup crash).
+- An Instance is gated until its quota is actually granted, and ungated promptly once it
+  is.
+
+### Non-Goals
+
+- WorkloadDeployment-level quota (this covers Instance-level claims only).
+- A live claim watch at the edge (deferred — see [Alternatives](#alternatives)).
+- Cleanup or migration of claims written to the wrong cluster before the shipped fix.
+- Changes to the quota evaluator itself; this enhancement produces claims and observes
+  grants, it does not decide them.
+
+## Proposal
+
+The deployments differ in exactly one way that matters: **where an Instance's
+ResourceClaim lives**, and from that, how a grant is observed.
+
+| Deployment | Where ResourceClaims live | How quota is enforced |
+|---|---|---|
+| **Management plane** | Per-project Milo control planes the manager already engages | Live watch on those claims (shipped) |
+| **Edge cell** | Remote Milo — one control plane per project, not on the cell | Claim written to the owning project; grant observed via the always-on backoff reconcile — no live watch |
+| **Flat single-tenant** | Co-located on the one cluster | Live watch on the local claims |
+
+In every case a claim is created in, and read from, the control plane that owns the
+project's quota ledger — write and read must be the same ledger, or grants are never
+seen. Enforcement has an always-on backoff floor (a pending Instance's reconcile re-runs
+on a backing-off schedule and re-checks its claim); at the edge that floor is the sole
+liveness mechanism, and everywhere else it backstops the live watch.
+
+### User Stories
+
+- As an operator, I stand up compute-manager on an edge cell with quota enabled; it
+  starts cleanly, and customer Instances there are gated on grants from their projects'
+  quota ledgers back in the platform.
+- As an operator, I run a flat single-tenant cluster with quota co-located; Instances are
+  gated on grants from the local ledger, with the cluster's single project named in
+  configuration.
+
+### Notes/Constraints/Caveats
+
+- **Flat single-tenant identity.** A flat deployment may not run the platform layer that
+  derives a claim's project from namespace-ownership labels, so it names its single
+  project explicitly in configuration and attributes every claim to it.
+- **Cell-wide Instance name uniqueness (flat mode).** See the known limitation under
+  [Drawbacks](#drawbacks) — in flat mode Instance names must be unique across the whole
+  cell.
+
+### Risks and Mitigations
+
+- **Edge grant latency.** With no live edge watch, a grant is observed on the backoff
+  schedule (seconds, backing off), not instantly. Accepted as the tradeoff of deferring
+  the edge watch; correctness is unaffected (an Instance stays gated until granted).
+- **Missing edge access to the remote ledger.** If a cell can't reach its projects'
+  ledgers, its Instances stay gated until access is fixed — correct (never a bypass), but
+  it blocks scheduling and shows up as recurring reconcile errors rather than one obvious
+  failure.
+- **Flat-mode name collision.** Same-named Instances in different namespaces contend for
+  one claim; mitigated by the cell-wide-uniqueness rule until claim identity is
+  namespace-qualified.
+
+## Design Details
+
+### Where claims live decides how grants are observed
+
+A **live** claim watch runs only where the claims sit on a cluster the manager already
+engages — the management plane's per-project control planes, and a flat single-tenant
+cluster's local API. An edge cell's claims are remote and split per project, so it runs
+no live watch. That also removes the startup crash: the edge's local cluster does not
+serve the quota API and should never have been asked to watch that kind. The rule is
+simply: watch claims live only where a cluster the manager already engages serves the
+quota API; determining this must fail loud rather than silently resolve to "off."
+
+### Claims route to the owning ledger
+
+Creating, reading, and deleting an Instance's claim targets the control plane that owns
+the project's quota ledger — the project control plane for management plane and edge, the
+local cluster for flat single-tenant. At the edge this routing doubles as the liveness
+mechanism: each backoff reconcile creates-or-reads the remote claim and checks whether it
+has been granted.
+
+### Flat single-tenant identity
+
+Because a flat deployment may lack the platform namespace-ownership layer, it takes its
+project identity from configuration and attributes every claim to that one project. If
+local quota is enabled without a configured project, the deployment refuses to start
+rather than emit claims the evaluator can't attribute.
+
+### Access prerequisites
+
+Each deployment needs only the access for the operations it performs, granted
+platform-side (not from this repository):
+
+- **Edge cell:** create/read/delete on the remote per-project claims — but no list/watch,
+  since it runs no live watch, and no project-discovery permission.
+- **Flat single-tenant:** access to the co-located quota API on its own cluster.
+- **Management plane:** the existing provider-cluster watch is unchanged.
+
+## Production Readiness Review Questionnaire
+
+<!-- Trimmed to the subsections relevant at alpha, following the precedent of other
+compute enhancements. Beta-targeted subsections (rollout/upgrade/rollback planning) are
+deferred. -->
+
+### Feature Enablement and Rollback
+
+- [ ] Feature gate
+- [x] Other
+  - Enablement is by deployment configuration, not a global gate: a deployment enforces
+    quota when it is configured with access to a quota ledger (and, for flat
+    single-tenant, a configured project). No configuration means quota is not enforced.
+  - Enabling/disabling requires no control-plane downtime; it takes effect on
+    compute-manager restart with the changed configuration.
+  - Rollback is removing the quota configuration; enforcement stops and Instances are no
+    longer gated on grants.
+
+### Monitoring Requirements
+
+- [x] API .status
+  - The Instance's quota-granted status condition tells an operator whether an Instance is
+    gated or granted. An Instance stuck ungated indicates quota is pending or a ledger it
+    can't reach.
+
+### Dependencies
+
+- Milo quota evaluator and the per-project (or local) quota ledgers it serves — the
+  authority that grants or denies claims. If unavailable, claims are not granted and
+  Instances stay gated (no bypass).
+- Platform-side access grants (see [Access prerequisites](#access-prerequisites)).
+
+**Open verification item.** It is not yet confirmed that Milo's quota evaluator actually
+grants a flat single-tenant claim attributed to the configured project — i.e. that the
+project is entitled in a co-located ledger. This is platform-side behavior, testable only
+in a running deployment, and must be verified before relying on flat single-tenant
+enforcement.
+
+### Scalability
+
+- The edge path adds no live watch, so it introduces no per-project watch fan-out; its
+  only added calls are the per-Instance claim create/read on the backoff schedule.
+
+### Troubleshooting
+
+- **Instances stuck ungated on an edge cell:** confirm the cell's create/read access to
+  its projects' remote ledgers, and that those projects are entitled so claims are
+  granted rather than left pending.
+- **Instances stuck ungated in flat single-tenant:** confirm a project is configured and
+  that the local evaluator grants claims attributed to it (see the open verification item
+  under [Dependencies](#dependencies)).
+
+## Implementation History
+
+- Management-plane claim routing and the live watch on project control planes:
+  **implemented** (verified 2026-05-26).
+- Edge cell (no live watch, backoff-only) and flat single-tenant (co-located watch,
+  configured project): **proposed** here; edge fixes the startup crash.
+
+## Drawbacks
+
+- Edge enforcement observes grants on the backoff schedule, not instantly — a latency
+  cost relative to a live watch.
+- Flat single-tenant carries an operator constraint until claim identity is
+  namespace-qualified: **Instance names must be unique across the whole cell**, because a
+  claim's identity derives from its Instance's name and all of a project's claims share
+  one space, so two same-named Instances in different namespaces would collide on one
+  claim.
+
+## Alternatives
+
+- **A live edge watch via a per-project-control-plane forwarder** — *deferred*. It means
+  one watch per project control plane a cell serves (fan-out that grows with projects),
+  and the always-on backoff already delivers grants within seconds, so the machinery
+  isn't worth it now. Revisit if edge grant latency ever becomes a product concern.
+- **A single unified multi-cluster provider spanning the cell and the remote ledger** —
+  rejected; it destabilizes the cell's control path by breaking the single-cluster
+  assumption its controllers rely on.
+- **Installing the quota API locally on edge cells** — rejected; the edge should read the
+  authoritative ledger, not host it.
+
+## Infrastructure Needed
+
+- Platform-side access grants in the project (or local) control planes, per
+  [Access prerequisites](#access-prerequisites). These cannot be granted from the compute
+  repository and should be tracked via the enhancement's tracking issue.