Phase 0 of the new SpotSchedule feature: ADR and CALM diagram update.

.claude/CHANGELOG.md

@@ -9,6 +9,54 @@ The format is based on the regulated environment requirements:
 
 ---
 
+## [2026-06-13 14:00] - ADR 0006/0007 + CALM: pluggable spot-schedule provider contract (Phase 0)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `docs/adr/0006-pluggable-spot-schedule-provider-contract.md`: NEW — accept a
+  duck-typed external provider contract. `ScheduledMachine.spec.spotSchedule`
+  (`apiVersion`/`kind`/`name`, same namespace) delegates the active/inactive
+  decision to a CRD in the new `spotschedules.5spot.finos.org` group; 5-Spot
+  reads only `status.active` (+ `Ready`), never the provider spec, never writes
+  it. `spec.schedule` becomes optional (CEL: at least one of schedule/
+  spotSchedule); both set ⇒ logical AND; `killSwitch` overrides. Unresolved
+  references hold last-known state (fail-inactive when never resolved) and
+  surface `SpotScheduleResolved=False` — no flapping. Event-driven dynamic
+  per-GVK watch + reverse index. Providers are untrusted inputs (read-only,
+  group-pinned RBAC; same-namespace only).
+- `docs/adr/0007-crd-multi-version-and-conversion.md`: NEW — CRDs are
+  multi-version-capable (one Rust struct per served version, single
+  `storage: true`), `conversion.strategy: None` with additive-only evolution;
+  resolvers/watchers key off group+kind, never a pinned `apiVersion`. The first
+  breaking change is the trigger for a webhook (superseding ADR). No CALM impact
+  (generation/versioning policy). Applies to `ScheduledMachine` and the new
+  provider group.
+- `docs/adr/README.md`: index rows for ADR 0006 and 0007.
+- `docs/architecture/calm/architecture.json`: NEW node
+  `service-spot-schedule-provider` (external, untrusted) + data-asset
+  `data-asset-spot-schedule-cr`; relationships `rel-spot-schedule-cr-stored-in-api`,
+  `rel-provider-writes-spot-schedule-status`, `rel-controller-spot-schedule-watch`
+  (read-only, group-pinned least-privilege control); flow
+  `flow-spot-schedule-activation`. `make calm-validate` clean (0/0/0);
+  `make calm-diagrams` re-rendered `docs/src/architecture/{system,flows}.md`.
+- `docs/src/reference/spot-schedule-contract.md`: NEW — provider contract
+  skeleton (reference, status table, AND composition, unresolved behavior,
+  versioning policy); added to `docs/mkdocs.yml` Reference nav.
+
+### Why
+Phase 0 (ADD gate) of the spot-schedule provider roadmap: record and visualize
+the architecture before any code, so activation semantics (capital-markets
+calendars, PromQL emitters, change-freeze gates) can be implemented out-of-tree
+against a stable, versioned public contract. Team-approved to proceed
+2026-06-13.
+
+### Impact
+- [ ] Breaking change
+- [ ] Requires cluster rollout
+- [ ] Config change only
+- [x] Documentation only
+
 ## [2026-06-10 11:30] - ADR 0005: remove spec.kata.destPath — fixed host path, H-1 gate closed
 
 **Author:** Erick Bourgeois

docs/adr/0006-pluggable-spot-schedule-provider-contract.md

@@ -0,0 +1,184 @@
+<!--
+Copyright (c) 2026 Erick Bourgeois, 5-Spot
+SPDX-License-Identifier: Apache-2.0
+-->
+# 0006 — Pluggable spot-schedule provider contract via `spec.spotSchedule` and the `spotschedules.5spot.finos.org` API group
+
+- **Status:** Accepted
+- **Date:** 2026-06-13
+- **Deciders:** Erick Bourgeois, 5-Spot team
+- **Supersedes:** —
+- **Related:** ADR [0007](./0007-crd-multi-version-and-conversion.md) (the
+  multi-version/conversion story this contract depends on);
+  `ScheduleSpec` + `evaluate_schedule` (`src/crd.rs`, `src/reconcilers/helpers.rs`);
+  `ObjectReference`-style refs already in `src/crd.rs`;
+  the reclaim-agent Node-watch (`rel-controller-workload-kube-api`) as the
+  event-driven watch shape to mirror; contract page
+  `docs/src/reference/spot-schedule-contract.md`.
+
+## Context
+
+A `ScheduledMachine` decides "should this machine exist right now?" solely from
+the inline `spec.schedule` (`daysOfWeek` / `hoursOfDay` / `timezone` /
+`enabled`), evaluated by `evaluate_schedule()` in `src/reconcilers/helpers.rs`.
+That model is **closed**: every new activation semantic — exchange calendars,
+statutory holidays and early closes, change-freeze windows, metric-driven
+scale, a manual trading-desk drain — would force a new field on 5-Spot's CRD
+and new branches in its reconciler. The motivating case is capital markets:
+trading-floor compute must follow the *exchange session calendar*, which no
+`daysOfWeek`/`hoursOfDay` expression can encode.
+
+**Options weighed:**
+
+1. **Grow `spec.schedule`** with calendar/holiday/PromQL sub-objects. Rejected:
+   unbounded surface area, 5-Spot would own every scheduling dialect forever,
+   and operators still couldn't express semantics we didn't anticipate.
+2. **A plugin/script hook** (WASM, embedded expression language) evaluated
+   in-process. Rejected: a sandbox and execution-safety burden inside a
+   privileged controller, and still not a Kubernetes-native contract.
+3. **A duck-typed external provider resource** that owns the active/inactive
+   decision; 5-Spot consumes it generically. **Chosen.** This is the exact
+   shape Cluster API uses for `infrastructureRef` / `bootstrap.configRef`
+   (a contract over `apiVersion`/`kind`/`name` reading a well-known status
+   field) — a pattern 5-Spot already participates in from the *other* side, so
+   it is idiomatic here and familiar to operators.
+
+The cost of option 3 is the one recorded below: a provider is an **untrusted
+input** that can start and stop the machines that reference it, and an
+unresolved/late/unhealthy provider must never flap machines.
+
+## Decision
+
+### 1. `spec.spotSchedule` reference on `ScheduledMachine`
+
+`src/crd.rs` is the source of truth; CRD YAML and API docs are regenerated
+(`regen-crds` → `regen-api-docs`), never hand-edited.
+
+```rust
+// on ScheduledMachineSpec — schedule becomes optional (see §3)
+#[serde(skip_serializing_if = "Option::is_none")]
+pub spot_schedule: Option<SpotScheduleRef>,
+
+pub struct SpotScheduleRef {
+    pub api_version: String, // "spotschedules.5spot.finos.org/<version>"
+    pub kind: String,        // e.g. "CapitalMarketsSchedule"
+    pub name: String,        // object in the SAME namespace as the SM
+}
+```
+
+`SpotScheduleRef` keeps the existing ref conventions (`deny_unknown_fields`,
+camelCase, schemars bounds). The referenced object **must live in the
+`ScheduledMachine`'s own namespace** — no `namespace` field, no cross-namespace
+references in this version.
+
+### 2. The provider group and the duck-typed contract
+
+The provider API group is **`spotschedules.5spot.finos.org`** — a sub-group of
+the existing `5spot.finos.org` group. A CEL `XValidation` on
+`spotSchedule.apiVersion` pins the **group** to exactly
+`spotschedules.5spot.finos.org` (any served version is accepted, per ADR 0007);
+references to any other group are rejected at admission.
+
+5-Spot reads a **duck-typed `status`** and never the provider `spec`:
+
+| Provider `status` field | Required | Meaning to 5-Spot |
+|---|---|---|
+| `active` (bool) | **yes** | the single source of truth: the machine should be up |
+| `conditions[type=Ready]` | recommended | provider health; `Ready=False` ⇒ **unresolved**, *not* inactive |
+| `observedGeneration` (int) | recommended | staleness detection |
+| `lastTransitionTime` (string) | recommended | observability / transition metrics |
+
+Providers implement `spec` however they like (exchange calendars, PromQL,
+cron, a plain `enabled` toggle). The contract is published in
+`docs/src/reference/spot-schedule-contract.md`.
+
+### 3. Composition with `spec.schedule` (AND), and `killSwitch` precedence
+
+`spec.schedule` becomes `Option<ScheduleSpec>`. A CEL `XValidation` requires
+**at least one** of `schedule` / `spotSchedule` (existing SMs all set
+`schedule`, so this is non-breaking). When **both** are present the machine is
+active **iff the inline time window AND the provider both say active** — this
+supports "the market is open **and** only 09:00–17:00 local". `killSwitch`
+continues to override everything: `killSwitch=true` ⇒ inactive regardless of
+schedule or provider. Precedence, highest first:
+
+```
+killSwitch  >  (schedule AND spotSchedule)  >  schedule-only / spotSchedule-only
+```
+
+### 4. Unresolved references never flap machines
+
+A reference is **Unresolved** when the provider CRD is not installed, the named
+object is absent, it has no `status.active`, or its `Ready` condition is
+`False`/missing. On Unresolved, 5-Spot:
+
+- sets a `SpotScheduleResolved=False` condition on the `ScheduledMachine` with
+  a precise reason (`ProviderCRDNotInstalled`, `ProviderNotFound`,
+  `StatusActiveMissing`, `ProviderNotReady`),
+- increments `fivespot_spot_schedule_resolution_errors_total`, and
+- **holds the last known resolved state** — it does **not** tear down a running
+  machine because its provider went briefly unreadable. If the reference has
+  **never** resolved, the fail-safe default is **inactive** (no machine is
+  created on the strength of a reference we cannot read).
+
+This "hold-last-state, fail-inactive-when-never-resolved" rule is the sharpest
+edge of the contract and is tested on both branches.
+
+### 5. Event-driven dynamic watch (no polling)
+
+The controller maintains an in-memory **reverse index**
+`(group, version-agnostic kind, namespace, name) → {ScheduledMachine keys}`,
+updated on SM apply/delete. For each distinct **GVK** referenced, it lazily
+starts a dynamic watch (`Api<DynamicObject>` resolved via `kube::discovery`)
+and maps provider events back through the index to `reconcile_on` the affected
+`ScheduledMachine`s. Streams are stopped when the last referencing SM goes
+away. The index is **rebuilt from the SM list on controller start** — nothing
+is stored outside Kubernetes (5-Spot stays stateless). This mirrors the
+existing reclaim-agent Node-watch and honours the project's
+event-driven-not-polling rule; provider `status.active` transitions drive SM
+reconciles at watch latency, matching the inline-schedule path.
+
+Hard edges handled (and tested): provider CRD installed *after* SMs reference
+it (discovery retry with backoff, surfaced as `Unresolved` meanwhile), provider
+CRD deleted while watched (stream error → re-resolve), controller restart
+(index rebuilt on boot).
+
+### 6. Security boundary — providers are untrusted inputs
+
+- Controller RBAC gains **`get;list;watch` only** on
+  `spotschedules.5spot.finos.org` `*` — **no write** to any provider resource.
+- A reference provider's own controller gets `update;patch` on **its own**
+  kind's `/status` subresource only (a separate Role, not 5-Spot's ClusterRole).
+- A compromised or buggy provider can flap the machines that *reference* it —
+  the **same blast radius as a malicious edit to `spec.schedule.enabled`**, and
+  bounded to SMs that opted in by naming it. Mitigations: same-namespace-only
+  references, RBAC on the provider CRDs, audit via the `SpotScheduleResolved`
+  condition + transition metrics, and a transition-rate alert. This actor and
+  these mitigations are added to `docs/src/security/threat-model.md`.
+
+## Consequences
+
+- **Easier:** activation semantics evolve **out of tree** — capital-markets
+  calendars, PromQL emitters, change-freeze gates ship as provider CRDs without
+  touching 5-Spot's CRD or reconciler; 5-Spot gains **no** new write ability
+  (read-only on providers); the inline `spec.schedule` stays the simple path.
+- **Harder / new burden:** a dynamic multi-GVK watch manager + reverse index to
+  build and test; a published, versioned **public contract** (ADR 0007) we must
+  not break; provider-flapping is a new failure mode requiring debounce-style
+  thinking (a per-SM `minimumStateDuration` is noted as possible future work,
+  not adopted here).
+- **Operator contract:** the referenced provider object **must pre-exist** in
+  the SM's namespace and report `status.active`; absence/unreadiness is a
+  loud `SpotScheduleResolved=False` status, never a silent machine teardown.
+- **Ruled out (this version):** cross-namespace / cross-cluster references;
+  multiple `spotSchedule` refs per SM (no AND/OR ref-lists); 5-Spot ever
+  writing provider status; a provider SDK/conformance kit; the Prometheus
+  emitter implementation itself (a planned *consumer* of this contract).
+- **CALM impact:** **updated.** New external node
+  `service-spot-schedule-provider`; a `data-asset-spot-schedule-cr`; a
+  controller→provider **watch** relationship
+  (`rel-controller-spot-schedule-watch`, read-only, group-pinned, least
+  privilege control); and a flow `flow-spot-schedule-activation`
+  (provider `status.active` transition → SM reconcile → activate/deactivate via
+  the existing schedule flows). `make calm-validate` + `make calm-diagrams`
+  before implementation.

docs/adr/0007-crd-multi-version-and-conversion.md

@@ -0,0 +1,119 @@
+<!--
+Copyright (c) 2026 Erick Bourgeois, 5-Spot
+SPDX-License-Identifier: Apache-2.0
+-->
+# 0007 — CRD multi-version support with `None` conversion and additive-only evolution
+
+- **Status:** Accepted
+- **Date:** 2026-06-13
+- **Deciders:** Erick Bourgeois, 5-Spot team
+- **Supersedes:** —
+- **Related:** ADR [0006](./0006-pluggable-spot-schedule-provider-contract.md)
+  (introduces the public `spotschedules.5spot.finos.org` contract this policy
+  governs); `src/crd.rs` (`#[kube(...)]` derive, source of truth); `crdgen`
+  (`src/bin/crdgen.rs`); the `regen-crds` / `regen-api-docs` skills.
+
+## Context
+
+5-Spot's CRDs are served at a single version, `v1alpha1`, today
+(`5spot.finos.org/v1alpha1`, kind `ScheduledMachine`). ADR 0006 introduces a
+**second** API group, `spotschedules.5spot.finos.org`, whose `status.active`
+contract is **implemented by third parties** — it is a public API surface, not
+an internal type. Public contracts evolve: fields get added, semantics get
+clarified, an alpha graduates to beta to stable. We need a deliberate story for
+serving more than one version of a CRD **before** we generate the first one, so
+the contract can mature without breaking existing `ScheduledMachine`s or
+out-of-tree providers.
+
+The constraints that shape the decision:
+
+- **kube-rs derives one version per struct.** `#[kube(version = "…")]` on a
+  `CustomResource` derive emits exactly one version. Serving N versions means N
+  Rust structs and a **merged** CRD whose `spec.versions[]` lists all of them
+  with exactly **one** `storage: true`.
+- **Conversion needs either compatibility or a webhook.** Kubernetes converts
+  between served versions one of two ways: `conversion.strategy: None`, which
+  works **only** when every served version round-trips losslessly through the
+  stored object (the API server just relabels `apiVersion`); or a **conversion
+  webhook**, which can transform fields but requires a TLS-served HTTPS endpoint
+  plus a cert lifecycle (cert-manager / `caBundle` rotation).
+- **5-Spot runs no webhook server today.** The controller exposes only
+  `/metrics` and `/healthz`. Standing up an admission/conversion webhook is a
+  non-trivial new failure domain (TLS, availability, fail-open/closed posture).
+
+**Options weighed:**
+
+1. **Stay single-version, bump in place.** Rejected: breaks every stored object
+   and every out-of-tree provider the moment the contract changes; unacceptable
+   for a public API.
+2. **Multi-version with a conversion webhook from day one.** Rejected *for now*:
+   pays the full webhook/TLS/cert cost before any breaking change exists to
+   justify it.
+3. **Multi-version with `conversion.strategy: None` + additive-only
+   evolution.** **Chosen.** Serve multiple versions that are all structurally
+   round-trippable through the storage version; constrain changes to be
+   additive (new optional fields, never a rename/retype/removal of a served
+   field) so `None` conversion stays correct. The first genuinely *breaking*
+   change is the documented trigger to introduce a webhook via a superseding
+   ADR.
+
+## Decision
+
+1. **5-Spot CRDs are multi-version-capable.** Each served version of a CRD is a
+   distinct Rust struct in `src/crd.rs` (e.g. `ScheduledMachine` at
+   `v1alpha1`, a future `ScheduledMachineV1Beta1` at `v1beta1`), and `crdgen`
+   emits a single CRD object per kind whose `spec.versions[]` lists every served
+   version with exactly **one** marked `storage: true`. This policy applies to
+   **both** `5spot.finos.org/ScheduledMachine` and the new
+   `spotschedules.5spot.finos.org` provider group from ADR 0006.
+
+2. **Conversion strategy is `None`; evolution is additive-only.** Generated
+   CRDs set `spec.conversion.strategy: None`. Because the API server performs
+   no field transformation under `None`, every served version **must** be
+   round-trippable through the storage schema. We therefore constrain
+   cross-version changes to be **additive** — new **optional** fields only;
+   never rename, retype, remove, or change the meaning of a field that an older
+   served version exposes. Defaulting is handled in the Rust types / reconciler,
+   not by conversion.
+
+3. **Resolvers and watchers are version-agnostic.** No code keys off a single
+   hardcoded `apiVersion` string. The spot-schedule resolver and dynamic watch
+   (ADR 0006 §5) key off **`group` + `kind`** and accept any *served* version of
+   a provider; the CEL pin on `spotSchedule.apiVersion` validates the **group**,
+   not a specific version.
+
+4. **Generation and CI guards.** `crdgen` produces multi-version YAML; serde
+   round-trip tests cover **every served version** of every kind; a CI/test
+   guard asserts the **single-`storage: true`** invariant per CRD. CRD shape
+   changes still start in `src/crd.rs` and flow through `regen-crds` →
+   `regen-api-docs` (LAST), never hand-edited YAML.
+
+5. **The webhook trigger is recorded, not built.** The first change that cannot
+   be expressed additively (a true breaking change to a served version) is the
+   trigger to introduce a conversion webhook. That is a **new architectural
+   decision** — it gets its own ADR superseding the relevant part of this one,
+   covering the webhook's TLS/cert lifecycle and fail posture. Until then,
+   5-Spot ships **no** webhook.
+
+## Consequences
+
+- **Easier:** the public `spotschedules` contract (and `ScheduledMachine`
+  itself) can grow new versions without breaking stored objects or out-of-tree
+  providers, and without 5-Spot standing up a webhook server — no TLS, cert, or
+  webhook-availability burden while the contract is young.
+- **Harder / constraints accepted:** cross-version changes are **disciplined to
+  additive-only**; a rename/retype/removal is *not* a routine change — it forces
+  the webhook ADR. Reviewers must police this on every CRD edit. Multiple Rust
+  structs per kind add some duplication, and round-trip tests multiply by served
+  version count.
+- **Operational invariant:** exactly one version per CRD is `storage: true`;
+  the CI guard fails the build otherwise. Reading/writing always converges on
+  the storage version with no lossy transformation.
+- **Ruled out (for now):** conversion webhooks; non-additive version changes;
+  version-pinned resolver/watch code. Each is reintroduced only via a
+  superseding ADR.
+- **CALM impact:** **none.** This is a generation/versioning policy for the CRD
+  artifacts and the type layer — it changes neither the running system's
+  topology (controller, agents, CAPI/provider resources, flows) nor any
+  trust boundary. CALM models the system, not the CRD-version-evolution policy,
+  so there is no node/relationship/flow to add.