diff --git a/.claude/CHANGELOG.md b/.claude/CHANGELOG.md index be67673..81b0525 100644 --- a/.claude/CHANGELOG.md +++ b/.claude/CHANGELOG.md @@ -9,6 +9,54 @@ The format is based on the regulated environment requirements: --- +## [2026-06-13 14:00] - ADR 0006/0007 + CALM: pluggable spot-schedule provider contract (Phase 0) + +**Author:** Erick Bourgeois + +### Changed +- `docs/adr/0006-pluggable-spot-schedule-provider-contract.md`: NEW — accept a + duck-typed external provider contract. `ScheduledMachine.spec.spotSchedule` + (`apiVersion`/`kind`/`name`, same namespace) delegates the active/inactive + decision to a CRD in the new `spotschedules.5spot.finos.org` group; 5-Spot + reads only `status.active` (+ `Ready`), never the provider spec, never writes + it. `spec.schedule` becomes optional (CEL: at least one of schedule/ + spotSchedule); both set ⇒ logical AND; `killSwitch` overrides. Unresolved + references hold last-known state (fail-inactive when never resolved) and + surface `SpotScheduleResolved=False` — no flapping. Event-driven dynamic + per-GVK watch + reverse index. Providers are untrusted inputs (read-only, + group-pinned RBAC; same-namespace only). +- `docs/adr/0007-crd-multi-version-and-conversion.md`: NEW — CRDs are + multi-version-capable (one Rust struct per served version, single + `storage: true`), `conversion.strategy: None` with additive-only evolution; + resolvers/watchers key off group+kind, never a pinned `apiVersion`. The first + breaking change is the trigger for a webhook (superseding ADR). No CALM impact + (generation/versioning policy). Applies to `ScheduledMachine` and the new + provider group. +- `docs/adr/README.md`: index rows for ADR 0006 and 0007. +- `docs/architecture/calm/architecture.json`: NEW node + `service-spot-schedule-provider` (external, untrusted) + data-asset + `data-asset-spot-schedule-cr`; relationships `rel-spot-schedule-cr-stored-in-api`, + `rel-provider-writes-spot-schedule-status`, `rel-controller-spot-schedule-watch` + (read-only, group-pinned least-privilege control); flow + `flow-spot-schedule-activation`. `make calm-validate` clean (0/0/0); + `make calm-diagrams` re-rendered `docs/src/architecture/{system,flows}.md`. +- `docs/src/reference/spot-schedule-contract.md`: NEW — provider contract + skeleton (reference, status table, AND composition, unresolved behavior, + versioning policy); added to `docs/mkdocs.yml` Reference nav. + +### Why +Phase 0 (ADD gate) of the spot-schedule provider roadmap: record and visualize +the architecture before any code, so activation semantics (capital-markets +calendars, PromQL emitters, change-freeze gates) can be implemented out-of-tree +against a stable, versioned public contract. Team-approved to proceed +2026-06-13. + +### Impact +- [ ] Breaking change +- [ ] Requires cluster rollout +- [ ] Config change only +- [x] Documentation only + ## [2026-06-10 11:30] - ADR 0005: remove spec.kata.destPath — fixed host path, H-1 gate closed **Author:** Erick Bourgeois diff --git a/docs/adr/0006-pluggable-spot-schedule-provider-contract.md b/docs/adr/0006-pluggable-spot-schedule-provider-contract.md new file mode 100644 index 0000000..1747e3d --- /dev/null +++ b/docs/adr/0006-pluggable-spot-schedule-provider-contract.md @@ -0,0 +1,184 @@ + +# 0006 — Pluggable spot-schedule provider contract via `spec.spotSchedule` and the `spotschedules.5spot.finos.org` API group + +- **Status:** Accepted +- **Date:** 2026-06-13 +- **Deciders:** Erick Bourgeois, 5-Spot team +- **Supersedes:** — +- **Related:** ADR [0007](./0007-crd-multi-version-and-conversion.md) (the + multi-version/conversion story this contract depends on); + `ScheduleSpec` + `evaluate_schedule` (`src/crd.rs`, `src/reconcilers/helpers.rs`); + `ObjectReference`-style refs already in `src/crd.rs`; + the reclaim-agent Node-watch (`rel-controller-workload-kube-api`) as the + event-driven watch shape to mirror; contract page + `docs/src/reference/spot-schedule-contract.md`. + +## Context + +A `ScheduledMachine` decides "should this machine exist right now?" solely from +the inline `spec.schedule` (`daysOfWeek` / `hoursOfDay` / `timezone` / +`enabled`), evaluated by `evaluate_schedule()` in `src/reconcilers/helpers.rs`. +That model is **closed**: every new activation semantic — exchange calendars, +statutory holidays and early closes, change-freeze windows, metric-driven +scale, a manual trading-desk drain — would force a new field on 5-Spot's CRD +and new branches in its reconciler. The motivating case is capital markets: +trading-floor compute must follow the *exchange session calendar*, which no +`daysOfWeek`/`hoursOfDay` expression can encode. + +**Options weighed:** + +1. **Grow `spec.schedule`** with calendar/holiday/PromQL sub-objects. Rejected: + unbounded surface area, 5-Spot would own every scheduling dialect forever, + and operators still couldn't express semantics we didn't anticipate. +2. **A plugin/script hook** (WASM, embedded expression language) evaluated + in-process. Rejected: a sandbox and execution-safety burden inside a + privileged controller, and still not a Kubernetes-native contract. +3. **A duck-typed external provider resource** that owns the active/inactive + decision; 5-Spot consumes it generically. **Chosen.** This is the exact + shape Cluster API uses for `infrastructureRef` / `bootstrap.configRef` + (a contract over `apiVersion`/`kind`/`name` reading a well-known status + field) — a pattern 5-Spot already participates in from the *other* side, so + it is idiomatic here and familiar to operators. + +The cost of option 3 is the one recorded below: a provider is an **untrusted +input** that can start and stop the machines that reference it, and an +unresolved/late/unhealthy provider must never flap machines. + +## Decision + +### 1. `spec.spotSchedule` reference on `ScheduledMachine` + +`src/crd.rs` is the source of truth; CRD YAML and API docs are regenerated +(`regen-crds` → `regen-api-docs`), never hand-edited. + +```rust +// on ScheduledMachineSpec — schedule becomes optional (see §3) +#[serde(skip_serializing_if = "Option::is_none")] +pub spot_schedule: Option, + +pub struct SpotScheduleRef { + pub api_version: String, // "spotschedules.5spot.finos.org/" + pub kind: String, // e.g. "CapitalMarketsSchedule" + pub name: String, // object in the SAME namespace as the SM +} +``` + +`SpotScheduleRef` keeps the existing ref conventions (`deny_unknown_fields`, +camelCase, schemars bounds). The referenced object **must live in the +`ScheduledMachine`'s own namespace** — no `namespace` field, no cross-namespace +references in this version. + +### 2. The provider group and the duck-typed contract + +The provider API group is **`spotschedules.5spot.finos.org`** — a sub-group of +the existing `5spot.finos.org` group. A CEL `XValidation` on +`spotSchedule.apiVersion` pins the **group** to exactly +`spotschedules.5spot.finos.org` (any served version is accepted, per ADR 0007); +references to any other group are rejected at admission. + +5-Spot reads a **duck-typed `status`** and never the provider `spec`: + +| Provider `status` field | Required | Meaning to 5-Spot | +|---|---|---| +| `active` (bool) | **yes** | the single source of truth: the machine should be up | +| `conditions[type=Ready]` | recommended | provider health; `Ready=False` ⇒ **unresolved**, *not* inactive | +| `observedGeneration` (int) | recommended | staleness detection | +| `lastTransitionTime` (string) | recommended | observability / transition metrics | + +Providers implement `spec` however they like (exchange calendars, PromQL, +cron, a plain `enabled` toggle). The contract is published in +`docs/src/reference/spot-schedule-contract.md`. + +### 3. Composition with `spec.schedule` (AND), and `killSwitch` precedence + +`spec.schedule` becomes `Option`. A CEL `XValidation` requires +**at least one** of `schedule` / `spotSchedule` (existing SMs all set +`schedule`, so this is non-breaking). When **both** are present the machine is +active **iff the inline time window AND the provider both say active** — this +supports "the market is open **and** only 09:00–17:00 local". `killSwitch` +continues to override everything: `killSwitch=true` ⇒ inactive regardless of +schedule or provider. Precedence, highest first: + +``` +killSwitch > (schedule AND spotSchedule) > schedule-only / spotSchedule-only +``` + +### 4. Unresolved references never flap machines + +A reference is **Unresolved** when the provider CRD is not installed, the named +object is absent, it has no `status.active`, or its `Ready` condition is +`False`/missing. On Unresolved, 5-Spot: + +- sets a `SpotScheduleResolved=False` condition on the `ScheduledMachine` with + a precise reason (`ProviderCRDNotInstalled`, `ProviderNotFound`, + `StatusActiveMissing`, `ProviderNotReady`), +- increments `fivespot_spot_schedule_resolution_errors_total`, and +- **holds the last known resolved state** — it does **not** tear down a running + machine because its provider went briefly unreadable. If the reference has + **never** resolved, the fail-safe default is **inactive** (no machine is + created on the strength of a reference we cannot read). + +This "hold-last-state, fail-inactive-when-never-resolved" rule is the sharpest +edge of the contract and is tested on both branches. + +### 5. Event-driven dynamic watch (no polling) + +The controller maintains an in-memory **reverse index** +`(group, version-agnostic kind, namespace, name) → {ScheduledMachine keys}`, +updated on SM apply/delete. For each distinct **GVK** referenced, it lazily +starts a dynamic watch (`Api` resolved via `kube::discovery`) +and maps provider events back through the index to `reconcile_on` the affected +`ScheduledMachine`s. Streams are stopped when the last referencing SM goes +away. The index is **rebuilt from the SM list on controller start** — nothing +is stored outside Kubernetes (5-Spot stays stateless). This mirrors the +existing reclaim-agent Node-watch and honours the project's +event-driven-not-polling rule; provider `status.active` transitions drive SM +reconciles at watch latency, matching the inline-schedule path. + +Hard edges handled (and tested): provider CRD installed *after* SMs reference +it (discovery retry with backoff, surfaced as `Unresolved` meanwhile), provider +CRD deleted while watched (stream error → re-resolve), controller restart +(index rebuilt on boot). + +### 6. Security boundary — providers are untrusted inputs + +- Controller RBAC gains **`get;list;watch` only** on + `spotschedules.5spot.finos.org` `*` — **no write** to any provider resource. +- A reference provider's own controller gets `update;patch` on **its own** + kind's `/status` subresource only (a separate Role, not 5-Spot's ClusterRole). +- A compromised or buggy provider can flap the machines that *reference* it — + the **same blast radius as a malicious edit to `spec.schedule.enabled`**, and + bounded to SMs that opted in by naming it. Mitigations: same-namespace-only + references, RBAC on the provider CRDs, audit via the `SpotScheduleResolved` + condition + transition metrics, and a transition-rate alert. This actor and + these mitigations are added to `docs/src/security/threat-model.md`. + +## Consequences + +- **Easier:** activation semantics evolve **out of tree** — capital-markets + calendars, PromQL emitters, change-freeze gates ship as provider CRDs without + touching 5-Spot's CRD or reconciler; 5-Spot gains **no** new write ability + (read-only on providers); the inline `spec.schedule` stays the simple path. +- **Harder / new burden:** a dynamic multi-GVK watch manager + reverse index to + build and test; a published, versioned **public contract** (ADR 0007) we must + not break; provider-flapping is a new failure mode requiring debounce-style + thinking (a per-SM `minimumStateDuration` is noted as possible future work, + not adopted here). +- **Operator contract:** the referenced provider object **must pre-exist** in + the SM's namespace and report `status.active`; absence/unreadiness is a + loud `SpotScheduleResolved=False` status, never a silent machine teardown. +- **Ruled out (this version):** cross-namespace / cross-cluster references; + multiple `spotSchedule` refs per SM (no AND/OR ref-lists); 5-Spot ever + writing provider status; a provider SDK/conformance kit; the Prometheus + emitter implementation itself (a planned *consumer* of this contract). +- **CALM impact:** **updated.** New external node + `service-spot-schedule-provider`; a `data-asset-spot-schedule-cr`; a + controller→provider **watch** relationship + (`rel-controller-spot-schedule-watch`, read-only, group-pinned, least + privilege control); and a flow `flow-spot-schedule-activation` + (provider `status.active` transition → SM reconcile → activate/deactivate via + the existing schedule flows). `make calm-validate` + `make calm-diagrams` + before implementation. diff --git a/docs/adr/0007-crd-multi-version-and-conversion.md b/docs/adr/0007-crd-multi-version-and-conversion.md new file mode 100644 index 0000000..861b17f --- /dev/null +++ b/docs/adr/0007-crd-multi-version-and-conversion.md @@ -0,0 +1,119 @@ + +# 0007 — CRD multi-version support with `None` conversion and additive-only evolution + +- **Status:** Accepted +- **Date:** 2026-06-13 +- **Deciders:** Erick Bourgeois, 5-Spot team +- **Supersedes:** — +- **Related:** ADR [0006](./0006-pluggable-spot-schedule-provider-contract.md) + (introduces the public `spotschedules.5spot.finos.org` contract this policy + governs); `src/crd.rs` (`#[kube(...)]` derive, source of truth); `crdgen` + (`src/bin/crdgen.rs`); the `regen-crds` / `regen-api-docs` skills. + +## Context + +5-Spot's CRDs are served at a single version, `v1alpha1`, today +(`5spot.finos.org/v1alpha1`, kind `ScheduledMachine`). ADR 0006 introduces a +**second** API group, `spotschedules.5spot.finos.org`, whose `status.active` +contract is **implemented by third parties** — it is a public API surface, not +an internal type. Public contracts evolve: fields get added, semantics get +clarified, an alpha graduates to beta to stable. We need a deliberate story for +serving more than one version of a CRD **before** we generate the first one, so +the contract can mature without breaking existing `ScheduledMachine`s or +out-of-tree providers. + +The constraints that shape the decision: + +- **kube-rs derives one version per struct.** `#[kube(version = "…")]` on a + `CustomResource` derive emits exactly one version. Serving N versions means N + Rust structs and a **merged** CRD whose `spec.versions[]` lists all of them + with exactly **one** `storage: true`. +- **Conversion needs either compatibility or a webhook.** Kubernetes converts + between served versions one of two ways: `conversion.strategy: None`, which + works **only** when every served version round-trips losslessly through the + stored object (the API server just relabels `apiVersion`); or a **conversion + webhook**, which can transform fields but requires a TLS-served HTTPS endpoint + plus a cert lifecycle (cert-manager / `caBundle` rotation). +- **5-Spot runs no webhook server today.** The controller exposes only + `/metrics` and `/healthz`. Standing up an admission/conversion webhook is a + non-trivial new failure domain (TLS, availability, fail-open/closed posture). + +**Options weighed:** + +1. **Stay single-version, bump in place.** Rejected: breaks every stored object + and every out-of-tree provider the moment the contract changes; unacceptable + for a public API. +2. **Multi-version with a conversion webhook from day one.** Rejected *for now*: + pays the full webhook/TLS/cert cost before any breaking change exists to + justify it. +3. **Multi-version with `conversion.strategy: None` + additive-only + evolution.** **Chosen.** Serve multiple versions that are all structurally + round-trippable through the storage version; constrain changes to be + additive (new optional fields, never a rename/retype/removal of a served + field) so `None` conversion stays correct. The first genuinely *breaking* + change is the documented trigger to introduce a webhook via a superseding + ADR. + +## Decision + +1. **5-Spot CRDs are multi-version-capable.** Each served version of a CRD is a + distinct Rust struct in `src/crd.rs` (e.g. `ScheduledMachine` at + `v1alpha1`, a future `ScheduledMachineV1Beta1` at `v1beta1`), and `crdgen` + emits a single CRD object per kind whose `spec.versions[]` lists every served + version with exactly **one** marked `storage: true`. This policy applies to + **both** `5spot.finos.org/ScheduledMachine` and the new + `spotschedules.5spot.finos.org` provider group from ADR 0006. + +2. **Conversion strategy is `None`; evolution is additive-only.** Generated + CRDs set `spec.conversion.strategy: None`. Because the API server performs + no field transformation under `None`, every served version **must** be + round-trippable through the storage schema. We therefore constrain + cross-version changes to be **additive** — new **optional** fields only; + never rename, retype, remove, or change the meaning of a field that an older + served version exposes. Defaulting is handled in the Rust types / reconciler, + not by conversion. + +3. **Resolvers and watchers are version-agnostic.** No code keys off a single + hardcoded `apiVersion` string. The spot-schedule resolver and dynamic watch + (ADR 0006 §5) key off **`group` + `kind`** and accept any *served* version of + a provider; the CEL pin on `spotSchedule.apiVersion` validates the **group**, + not a specific version. + +4. **Generation and CI guards.** `crdgen` produces multi-version YAML; serde + round-trip tests cover **every served version** of every kind; a CI/test + guard asserts the **single-`storage: true`** invariant per CRD. CRD shape + changes still start in `src/crd.rs` and flow through `regen-crds` → + `regen-api-docs` (LAST), never hand-edited YAML. + +5. **The webhook trigger is recorded, not built.** The first change that cannot + be expressed additively (a true breaking change to a served version) is the + trigger to introduce a conversion webhook. That is a **new architectural + decision** — it gets its own ADR superseding the relevant part of this one, + covering the webhook's TLS/cert lifecycle and fail posture. Until then, + 5-Spot ships **no** webhook. + +## Consequences + +- **Easier:** the public `spotschedules` contract (and `ScheduledMachine` + itself) can grow new versions without breaking stored objects or out-of-tree + providers, and without 5-Spot standing up a webhook server — no TLS, cert, or + webhook-availability burden while the contract is young. +- **Harder / constraints accepted:** cross-version changes are **disciplined to + additive-only**; a rename/retype/removal is *not* a routine change — it forces + the webhook ADR. Reviewers must police this on every CRD edit. Multiple Rust + structs per kind add some duplication, and round-trip tests multiply by served + version count. +- **Operational invariant:** exactly one version per CRD is `storage: true`; + the CI guard fails the build otherwise. Reading/writing always converges on + the storage version with no lossy transformation. +- **Ruled out (for now):** conversion webhooks; non-additive version changes; + version-pinned resolver/watch code. Each is reintroduced only via a + superseding ADR. +- **CALM impact:** **none.** This is a generation/versioning policy for the CRD + artifacts and the type layer — it changes neither the running system's + topology (controller, agents, CAPI/provider resources, flows) nor any + trust boundary. CALM models the system, not the CRD-version-evolution policy, + so there is no node/relationship/flow to add. diff --git a/docs/adr/README.md b/docs/adr/README.md index 4b9e986..cbae69a 100644 --- a/docs/adr/README.md +++ b/docs/adr/README.md @@ -41,3 +41,5 @@ refactors) need neither. When unsure, **write the ADR.** | [0003](./0003-in-pod-host-service-restart-via-nsenter.md) | In-pod host k0s-service restart via `nsenter` (privileged Kata-config agent) | Accepted | | [0004](./0004-agent-pod-security-exception-boundary-vap.md) | Agent pod-security exception boundary via deny-by-default ValidatingAdmissionPolicy | Accepted | | [0005](./0005-remove-kata-destpath-fixed-host-path.md) | Remove `spec.kata.destPath`; fix the host path to `/etc/k0s/containerd.d/kata.toml` | Accepted | +| [0006](./0006-pluggable-spot-schedule-provider-contract.md) | Pluggable spot-schedule provider contract via `spec.spotSchedule` and the `spotschedules.5spot.finos.org` API group | Accepted | +| [0007](./0007-crd-multi-version-and-conversion.md) | CRD multi-version support with `None` conversion and additive-only evolution | Accepted | diff --git a/docs/architecture/calm/architecture.json b/docs/architecture/calm/architecture.json index ca1e151..c5b832e 100644 --- a/docs/architecture/calm/architecture.json +++ b/docs/architecture/calm/architecture.json @@ -310,13 +310,40 @@ "node-type": "data-asset", "name": "Per-Node Kata Config ConfigMap", "description": "Source ConfigMap or Secret on the workload cluster, in spec.kata.namespace (default 5spot-system), holding the Kata containerd drop-in. Platform-/Flux-managed — 5-Spot never creates, writes, or deletes it. The controller resolves it read-only (existence check) and the kata config agent GETs it by exact name each poll tick, keyed by the 5spot.finos.org/kata-config-ref Node annotation. If absent, delivery fails fast with a status reason and the Node is not opted in (ADR 0002)." + }, + { + "unique-id": "service-spot-schedule-provider", + "node-type": "service", + "name": "Spot Schedule Provider (external)", + "description": "Third-party controller implementing the spotschedules.5spot.finos.org provider contract (ADR 0006). Reconciles a provider CRD (reference implementation: CapitalMarketsSchedule — exchange sessions, statutory holidays, early closes) and publishes a single duck-typed status.active boolean plus a Ready condition. 5-Spot treats it as an UNTRUSTED INPUT: it can start/stop the ScheduledMachines that reference it (same blast radius as editing spec.schedule.enabled), bounded to same-namespace SMs that named it. 5-Spot never reads the provider spec and never writes the provider object.", + "controls": { + "least-privilege": { + "description": "A reference provider's own controller holds update,patch on its OWN kind's /status subresource only (a separate Role, not 5-Spot's ClusterRole). It needs no access to ScheduledMachine or 5-Spot resources.", + "requirements": [ + { + "requirement-url": "https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final", + "config": { + "framework": "NIST SP 800-53 Rev. 5", + "controls": ["AC-3", "AC-6"], + "evidence": "docs/src/reference/spot-schedule-contract.md, docs/adr/0006-pluggable-spot-schedule-provider-contract.md" + } + } + ] + } + } + }, + { + "unique-id": "data-asset-spot-schedule-cr", + "node-type": "data-asset", + "name": "Spot Schedule Provider Custom Resource", + "description": "Namespaced CRD in the spotschedules.5spot.finos.org group (e.g. CapitalMarketsSchedule), stored in the management cluster API. Owned by the external provider controller, which writes status.active (the authoritative active/inactive signal), a Ready condition, observedGeneration, and lastTransitionTime. Referenced by ScheduledMachine.spec.spotSchedule (apiVersion/kind/name, same namespace). Multi-version per ADR 0007 (served versions, single storage version, conversion strategy None)." } ], "relationships": [ { "unique-id": "rel-mgmt-cluster-contains-controller", - "description": "The management cluster hosts the 5-Spot controller, the Kubernetes API, and the CAPI provider controllers.", + "description": "The management cluster hosts the 5-Spot controller, the Kubernetes API, the CAPI provider controllers, and any external spot-schedule provider controllers (ADR 0006).", "relationship-type": { "deployed-in": { "container": "ecosystem-management-cluster", @@ -325,7 +352,8 @@ "service-kubernetes-api", "service-capi-core", "service-capi-bootstrap-provider", - "service-capi-infrastructure-provider" + "service-capi-infrastructure-provider", + "service-spot-schedule-provider" ] } } @@ -649,6 +677,59 @@ "nodes": ["data-asset-kata-config-configmap"] } } + }, + { + "unique-id": "rel-spot-schedule-cr-stored-in-api", + "description": "Spot-schedule provider custom resources (spotschedules.5spot.finos.org) are persisted by the management cluster API server in etcd, in the referencing ScheduledMachine's namespace.", + "relationship-type": { + "composed-of": { + "container": "service-kubernetes-api", + "nodes": ["data-asset-spot-schedule-cr"] + } + } + }, + { + "unique-id": "rel-provider-writes-spot-schedule-status", + "description": "The external provider controller reconciles its own CRD and PATCHes status.active (plus a Ready condition, observedGeneration, lastTransitionTime) onto the spot-schedule custom resource via the management cluster API. It writes only its own kind's /status subresource.", + "relationship-type": { + "connects": { + "source": { "node": "service-spot-schedule-provider" }, + "destination": { + "node": "service-kubernetes-api", + "interfaces": ["kube-api-https"] + } + } + }, + "protocol": "HTTPS" + }, + { + "unique-id": "rel-controller-spot-schedule-watch", + "description": "Controller watches the spot-schedule provider custom resource named by ScheduledMachine.spec.spotSchedule and reads its duck-typed status.active (and Ready condition). Event-driven via a dynamic per-GVK watch (Api resolved through kube discovery) mapped back to referencing ScheduledMachines through an in-memory reverse index; provider status.active transitions trigger SM reconciles at watch latency. Read-only: 5-Spot never writes the provider object. ADR 0006 §5.", + "relationship-type": { + "connects": { + "source": { "node": "service-five-spot-controller" }, + "destination": { + "node": "service-kubernetes-api", + "interfaces": ["kube-api-https"] + } + } + }, + "protocol": "HTTPS", + "controls": { + "least-privilege-readonly-group-pinned": { + "description": "Controller RBAC grants get,list,watch ONLY on spotschedules.5spot.finos.org/* — no write to any provider resource. A CEL XValidation on spec.spotSchedule.apiVersion pins the group to exactly spotschedules.5spot.finos.org (any served version accepted, ADR 0007), rejecting references to any other group at admission. References are same-namespace only. Providers are untrusted inputs whose blast radius (flapping the SMs that reference them) equals editing spec.schedule.enabled; mitigations are audited via the SpotScheduleResolved condition and transition-rate metrics.", + "requirements": [ + { + "requirement-url": "https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final", + "config": { + "framework": "NIST SP 800-53 Rev. 5", + "controls": ["AC-3", "AC-4", "AC-6", "SI-10"], + "evidence": "docs/adr/0006-pluggable-spot-schedule-provider-contract.md, docs/src/security/threat-model.md" + } + } + ] + } + } } ], @@ -848,6 +929,49 @@ ] } } + }, + { + "unique-id": "flow-spot-schedule-activation", + "name": "Spot Schedule Activation (provider-driven active/inactive)", + "description": "A ScheduledMachine with spec.spotSchedule delegates its active/inactive decision to an external provider's duck-typed status.active. The controller watches the provider CR event-driven and composes the provider verdict with the inline spec.schedule (logical AND when both are set; killSwitch always wins). An unresolved/unready provider never flaps machines: the controller surfaces SpotScheduleResolved=False and holds last-known state (fail-inactive when never resolved). ADR 0006.", + "transitions": [ + { + "sequence-number": 1, + "relationship-unique-id": "rel-provider-writes-spot-schedule-status", + "description": "External provider controller evaluates its own spec (e.g. exchange calendar/holidays/early-closes) and PATCHes status.active (and the Ready condition) onto its CR. 5-Spot never reads the provider spec — only this status." + }, + { + "sequence-number": 2, + "relationship-unique-id": "rel-controller-spot-schedule-watch", + "description": "Controller's dynamic watch on the provider GVK fires; the reverse index maps the event to the referencing ScheduledMachine(s) and enqueues them. The resolver reads status.active and the Ready condition defensively (missing CRD/object/status or Ready=False ⇒ Unresolved, surfaced as SpotScheduleResolved=False, not inactive).", + "direction": "destination-to-source" + }, + { + "sequence-number": 3, + "relationship-unique-id": "rel-controller-kube-api", + "description": "Controller composes the provider verdict with evaluate_schedule (AND when spec.schedule is also set; killSwitch overrides). If the composite verdict is active and the machine is not yet up, it proceeds into flow-schedule-activation (create bootstrap/infra + CAPI Machine)." + }, + { + "sequence-number": 4, + "relationship-unique-id": "rel-controller-kube-api", + "description": "On a provider transition to status.active=false (composite verdict inactive), the controller proceeds into flow-schedule-deactivation (cordon, drain within nodeDrainTimeout, delete the CAPI Machine). Provider Unresolved does NOT trigger this — last-known state is held." + } + ], + "controls": { + "untrusted-provider-failsafe": { + "description": "Providers are untrusted inputs. Group-pinned read-only RBAC (get,list,watch on spotschedules.5spot.finos.org/* only), same-namespace-only references, and the CEL group pin bound the blast radius to SMs that named the provider. Unresolved references hold last-known state and fail inactive when never resolved, so a missing/unready/compromised provider degrades to a loud SpotScheduleResolved=False status rather than flapping machines. Transition-rate metrics + an alert detect flapping.", + "requirements": [ + { + "requirement-url": "https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final", + "config": { + "framework": "NIST SP 800-53 Rev. 5", + "controls": ["AC-4", "AC-6", "SI-10", "AU-2"], + "evidence": "docs/adr/0006-pluggable-spot-schedule-provider-contract.md, docs/src/security/threat-model.md" + } + } + ] + } + } } ], @@ -882,6 +1006,7 @@ "adrs": [ "../../adr/0002-kata-config-delivery-via-spec-kata.md", - "../../adr/0003-in-pod-host-service-restart-via-nsenter.md" + "../../adr/0003-in-pod-host-service-restart-via-nsenter.md", + "../../adr/0006-pluggable-spot-schedule-provider-contract.md" ] } diff --git a/docs/mkdocs.yml b/docs/mkdocs.yml index 1ba4b98..835e484 100644 --- a/docs/mkdocs.yml +++ b/docs/mkdocs.yml @@ -244,6 +244,7 @@ nav: - Reference: - API Reference: reference/api.md + - Spot Schedule Provider Contract: reference/spot-schedule-contract.md - CLI Reference: reference/cli.md - Changelog: changelog.md - License: license.md diff --git a/docs/src/reference/spot-schedule-contract.md b/docs/src/reference/spot-schedule-contract.md new file mode 100644 index 0000000..c9a1b42 --- /dev/null +++ b/docs/src/reference/spot-schedule-contract.md @@ -0,0 +1,141 @@ + +# Spot Schedule Provider Contract + +> **Status:** Draft skeleton (Phase 0 of the spot-schedule provider roadmap). +> The contract is **Accepted** in [ADR 0006](https://github.com/finos/5-spot) +> but the implementation lands across later phases. This page is the +> authoritative specification a provider author implements against. + +A **spot-schedule provider** is any Kubernetes custom resource in the +`spotschedules.5spot.finos.org` API group that tells 5-Spot whether a +[`ScheduledMachine`](api.md) should be **active** (its machine should exist) or +**inactive** (its machine should be torn down) right now. + +5-Spot owns *where the decision lives* (`status.active`); the provider owns +*how the decision is computed* — an exchange calendar, a PromQL expression, a +change-freeze window, or a plain manual toggle. 5-Spot **never reads the +provider `spec`** and **never writes the provider object**. + +This is the same duck-typed contract pattern Cluster API uses for +`infrastructureRef` / `bootstrap.configRef`. + +## Referencing a provider + +A `ScheduledMachine` opts in by referencing a provider object in **its own +namespace**: + +```yaml +apiVersion: 5spot.finos.org/v1alpha1 +kind: ScheduledMachine +metadata: + name: trading-floor-rack-7 + namespace: capital-markets +spec: + # spec.schedule is optional when spotSchedule is set (and vice versa); + # at least one is required. + spotSchedule: + apiVersion: spotschedules.5spot.finos.org/v1alpha1 + kind: CapitalMarketsSchedule + name: nyse-equities-session # must live in namespace: capital-markets +``` + +- `apiVersion` — the group **must** be `spotschedules.5spot.finos.org` + (CEL-enforced at admission; any *served* version is accepted, see + [Versioning](#versioning)). Other groups are rejected. +- `kind` — the provider kind (e.g. `CapitalMarketsSchedule`). +- `name` — the provider object, **same namespace** as the `ScheduledMachine`. + Cross-namespace references are not supported. + +## The status contract a provider must satisfy + +5-Spot reads only the provider's `status`: + +| `status` field | Required | Type | Meaning | +|---|:--:|---|---| +| `active` | **yes** | bool | **The decision.** `true` ⇒ the referencing machine should be up; `false` ⇒ it should be down. | +| `conditions[type=Ready]` | recommended | condition | Provider health. `Ready=False` (or absent) ⇒ 5-Spot treats the reference as **unresolved**, *not* inactive (see [Unresolved behavior](#unresolved-behavior)). | +| `observedGeneration` | recommended | int64 | The `metadata.generation` the status reflects; lets 5-Spot detect stale status. | +| `lastTransitionTime` | recommended | RFC 3339 string | When `active` last flipped; used for observability and flap detection. | + +A minimal conformant `status`: + +```yaml +status: + active: true + observedGeneration: 4 + lastTransitionTime: "2026-06-13T13:30:00Z" + conditions: + - type: Ready + status: "True" + reason: SessionOpen + message: "NYSE equities regular session open" + lastTransitionTime: "2026-06-13T13:30:00Z" + observedGeneration: 4 +``` + +> **TODO (Phase 2/6):** finalize the exact `conditions` schema 5-Spot keys off +> (reason vocabulary, whether `observedGeneration` staleness is enforced). + +## Composition with `spec.schedule` + +`spec.schedule` and `spec.spotSchedule` are independent; **at least one** must +be set. When **both** are set, the machine is active **only if both agree** — +logical **AND** (e.g. "the market is open *and* it is 09:00–17:00 local"). +`spec.killSwitch` always wins. Precedence, highest first: + +``` +killSwitch > (schedule AND spotSchedule) > schedule-only / spotSchedule-only +``` + +## Unresolved behavior + +A reference is **Unresolved** when the provider CRD is not installed, the named +object is absent, it exposes no `status.active`, or its `Ready` condition is +`False`/missing. On Unresolved, 5-Spot: + +- sets a `SpotScheduleResolved=False` condition on the `ScheduledMachine` with + a precise reason (`ProviderCRDNotInstalled`, `ProviderNotFound`, + `StatusActiveMissing`, `ProviderNotReady`), +- emits `fivespot_spot_schedule_resolution_errors_total`, and +- **holds the last known resolved state** — a running machine is **not** torn + down because its provider briefly went unreadable. If the reference has + **never** resolved, the fail-safe default is **inactive**. + +A provider that misbehaves (or is compromised) can only flap the machines that +**reference it** — the same blast radius as editing `spec.schedule.enabled` — +and only within its own namespace. See the +[threat model](../security/threat-model.md). + +## Versioning + +Per [ADR 0007](https://github.com/finos/5-spot), provider CRDs are +**multi-version-capable**: a CRD may serve several versions +(`v1alpha1`, later `v1beta1`/`v1`) with exactly one storage version and +`conversion.strategy: None`. 5-Spot resolves providers by **group + kind**, not +a pinned `apiVersion`, so any *served* version of a referenced provider works. + +Contract evolution is **additive-only** while conversion is `None`: new +**optional** status fields may be added; an existing status field is never +renamed, retyped, or removed without a superseding ADR (which would introduce a +conversion webhook). Providers should therefore tolerate unknown fields and not +rely on 5-Spot reading anything beyond the table above. + +## Reference provider: `CapitalMarketsSchedule` + +The in-repo reference implementation (roadmap Phase 5) reconciles a declarative +exchange calendar (sessions, statutory holidays, early closes, timezone) into +`status.active`, requeuing at the next session/holiday boundary (event-driven — +a single timed requeue at the calendar transition, not a poll loop). + +> **TODO (Phase 5/6):** link the `CapitalMarketsSchedule` API reference, the +> guide, and a worked minimal "hello-world" provider example here. + +## See also + +- [ADR 0006 — Pluggable spot-schedule provider contract](https://github.com/finos/5-spot) +- [ADR 0007 — CRD multi-version + conversion](https://github.com/finos/5-spot) +- [`ScheduledMachine` API reference](api.md) +- [Threat model](../security/threat-model.md)