Phase 2 and 3: true event-driven watch, a ScheduledMachine.spec.spotSchedule reference now actually drives the machine's active/inactive decision (pull-on-reconcile)

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

Author: ebourgeois

.claude/CHANGELOG.md

@@ -9,6 +9,106 @@ The format is based on the regulated environment requirements:
 
 ---
 
+## [2026-06-14 11:00] - Spot-schedule dynamic event-driven provider watch (Phase 3)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/reconcilers/spot_schedule_watch.rs`: NEW. `ReverseIndex` (pure:
+  `(gvk, ns, name)` → referencing `ScheduledMachine`s, with per-SM record so a
+  changed/removed `spec.spotSchedule` updates precisely) + `provider_key_for`.
+  `SpotScheduleWatchManager` owns the index and lazily runs **one
+  `kube::runtime::watcher` stream per referenced provider GVK** (event-driven,
+  not polling); `sync_watchers` reconciles the running watcher set against the
+  index's referenced GVKs (spawn new, abort orphaned). Per-GVK task resolves the
+  `ApiResource` via `discovery::pinned_kind` with back-off
+  (`SPOT_SCHEDULE_DISCOVERY_RETRY_SECS`, handles CRD-installed-late and
+  CRD-deleted-while-watched), maps each provider event back through the index,
+  and enqueues the referencing SMs.
+- `src/main.rs`: a standalone `ScheduledMachine` reflector feeds SM apply/delete
+  into the manager (index rebuilt from the reflector's initial list on restart —
+  nothing stored outside Kubernetes); a second `Controller::reconcile_on`
+  consumes the manager's mpsc so provider `status.active` flips trigger SM
+  reconciles at watch latency.
+- `src/constants.rs`: `SPOT_SCHEDULE_EVENT_CHANNEL_CAP`,
+  `SPOT_SCHEDULE_DISCOVERY_RETRY_SECS`.
+- `src/reconcilers/mod.rs`: export the watch types + `compose_should_be_active`.
+- Tests: `spot_schedule_watch_tests.rs` (13 — ReverseIndex register/replace/
+  deregister/lookup/gvks, `provider_key_for`, manager observe/forget watcher
+  lifecycle incl. shared-GVK and watcher-stop-on-last-deref);
+  `tests/integration_spot_schedule.rs` (4 — create→active, flip→inactive,
+  delete→unresolved+hold/fail-inactive across a mock kube API, and the watch
+  index→reconcile mapping). 658 lib tests green.
+- Docs: contract page status → Phase 3.
+
+### Why
+Phase 3 of the spot-schedule provider roadmap: remove the Phase-2 pull latency.
+Provider status transitions are now a reconcile **trigger** (a true watch over
+the `spotschedules.5spot.finos.org` group), not something re-checked only on the
+controller's own timer — satisfying the project's watch-not-poll rule. Realizes
+the `rel-controller-spot-schedule-watch` relationship + `flow-spot-schedule-activation`
+already modeled in CALM at Phase 0 (no ADR/CALM change).
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout (new controller image; RBAC for
+      `spotschedules.5spot.finos.org` get/list/watch lands in Phase 4)
+- [ ] Config change only
+- [ ] Documentation only
+
+## [2026-06-13 18:45] - Spot-schedule resolver + composition + status surface (Phase 2)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/reconcilers/spot_schedule.rs`: NEW resolver module. `SpotScheduleVerdict`
+  enum (`Active`/`Inactive`/`Unresolved`); pure `verdict_from_status` (duck-typed
+  `status.active` + `Ready` extraction); async `resolve_spot_schedule`
+  (`discovery::pinned_kind` + `get_opt`). Transient API errors propagate (retry);
+  CRD-not-installed / object-absent / status-missing / not-Ready are `Unresolved`
+  verdicts, never errors or panics. **Ready semantics finalized:** absent `Ready`
+  ⇒ `active` authoritative; present non-`True` `Ready` ⇒ unresolved (resolves an
+  internal inconsistency in ADR 0006 §4 / contract table — both updated).
+- `src/reconcilers/helpers.rs`: NEW `compose_should_be_active` — AND composition
+  of the inline schedule and the provider verdict, with hold-last-state on
+  Unresolved (fail-inactive when never resolved). `killSwitch` precedence is
+  upstream.
+- `src/crd.rs`: replaced Phase-1 `effective_schedule()` with `is_enabled()` (a
+  spotSchedule-only machine is always enabled; the provider governs activity).
+  NEW `SpotScheduleStatus` + `ScheduledMachineStatus.spot_schedule` field (the
+  durable hold-last-state surface: `resolved`/`active`/`reason`/`message`/
+  `providerGeneration`/`lastTransitionTime`).
+- `src/reconcilers/scheduled_machine.rs`: `reconcile_inner` now evaluates the
+  optional inline schedule, resolves the provider (pull-on-reconcile), composes
+  `should_be_active`, records metrics, and patches `status.spotSchedule`
+  (`persist_spot_schedule_status`, a `/status` merge patch that coexists with the
+  phase/condition writes). Handler `enabled` guards use `is_enabled()`.
+- `src/metrics.rs`: NEW `fivespot_spot_schedule_resolutions_total{namespace,kind,result}`,
+  `_resolution_errors_total{namespace,kind,reason}`,
+  `_transitions_total{namespace,kind}` (+ record fns; bounded cardinality, never
+  keyed by name).
+- `deploy/crds/scheduledmachine.yaml` regenerated (status.spotSchedule);
+  `docs/src/reference/api.md` regenerated.
+- Tests: `spot_schedule_tests.rs` (verdict truth table + 3 mock-client resolve
+  paths), `helpers_tests.rs` (12-case composition truth table),
+  `scheduled_machine_tests.rs` (persist `/status` patch), `metrics_tests.rs`,
+  `crd_tests.rs` (status round-trip + is_enabled). 645 lib tests green.
+- Docs: ADR 0006 + contract page `Ready` clarification; contract page status →
+  Phase 2; `docs/src/operations/monitoring.md` spot-schedule metrics table.
+
+### Why
+Phase 2 of the spot-schedule provider roadmap: make a `spec.spotSchedule`
+reference actually drive the machine's active/inactive decision (pull-on-reconcile),
+with the ADR-specified hold-last-state fail-safe and an observable status surface.
+The event-driven watch (Phase 3) and the reference-provider controller (Phase 5)
+build on this.
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout (apply regenerated CRDs + new controller image)
+- [ ] Config change only
+- [ ] Documentation only
+
 ## [2026-06-13 16:30] - Spot-schedule provider API: CRD scaffolding + ScheduledMachine v1beta1 (Phase 1)
 
 **Author:** Erick Bourgeois

deploy/crds/scheduledmachine.yaml

@@ -673,6 +673,47 @@ spec:
                   (Pending, ShuttingDown, Inactive, Disabled, Terminated, Error) is
                   reported as `False`.
                 type: boolean
+              spotSchedule:
+                description: |-
+                  Spot-schedule provider resolution state (ADR 0006), present only when
+                  `spec.spotSchedule` is set. Carries the last resolved/held provider
+                  `active` value (the input to hold-last-state composition), the
+                  resolution reason/message, and the provider's observed generation.
+                nullable: true
+                properties:
+                  active:
+                    description: |-
+                      Last known provider `status.active`. Held across an unresolved reconcile
+                      (hold-last-state); `None` until the provider first resolves.
+                    nullable: true
+                    type: boolean
+                  lastTransitionTime:
+                    description: When `active` last transitioned (RFC3339).
+                    nullable: true
+                    type: string
+                  message:
+                    description: Human-readable resolution detail.
+                    nullable: true
+                    type: string
+                  providerGeneration:
+                    description: The provider's `status.observedGeneration` at the last resolution.
+                    format: int64
+                    nullable: true
+                    type: integer
+                  reason:
+                    description: Machine-readable resolution reason (a `REASON_SPOT_SCHEDULE_*` value).
+                    nullable: true
+                    type: string
+                  resolved:
+                    description: |-
+                      Whether the referenced provider resolved into an authoritative verdict
+                      this reconcile. `false` mirrors a `SpotScheduleResolved=False`
+                      condition — the provider CRD is absent, the object is missing, it
+                      exposes no `status.active`, or it is not `Ready`.
+                    type: boolean
+                required:
+                - resolved
+                type: object
             type: object
         required:
         - spec
@@ -1204,6 +1245,47 @@ spec:
                   (Pending, ShuttingDown, Inactive, Disabled, Terminated, Error) is
                   reported as `False`.
                 type: boolean
+              spotSchedule:
+                description: |-
+                  Spot-schedule provider resolution state (ADR 0006), present only when
+                  `spec.spotSchedule` is set. Carries the last resolved/held provider
+                  `active` value (the input to hold-last-state composition), the
+                  resolution reason/message, and the provider's observed generation.
+                nullable: true
+                properties:
+                  active:
+                    description: |-
+                      Last known provider `status.active`. Held across an unresolved reconcile
+                      (hold-last-state); `None` until the provider first resolves.
+                    nullable: true
+                    type: boolean
+                  lastTransitionTime:
+                    description: When `active` last transitioned (RFC3339).
+                    nullable: true
+                    type: string
+                  message:
+                    description: Human-readable resolution detail.
+                    nullable: true
+                    type: string
+                  providerGeneration:
+                    description: The provider's `status.observedGeneration` at the last resolution.
+                    format: int64
+                    nullable: true
+                    type: integer
+                  reason:
+                    description: Machine-readable resolution reason (a `REASON_SPOT_SCHEDULE_*` value).
+                    nullable: true
+                    type: string
+                  resolved:
+                    description: |-
+                      Whether the referenced provider resolved into an authoritative verdict
+                      this reconcile. `false` mirrors a `SpotScheduleResolved=False`
+                      condition — the provider CRD is absent, the object is missing, it
+                      exposes no `status.active`, or it is not `Ready`.
+                    type: boolean
+                required:
+                - resolved
+                type: object
             type: object
         required:
         - spec

docs/adr/0006-pluggable-spot-schedule-provider-contract.md

@@ -109,8 +109,12 @@ killSwitch  >  (schedule AND spotSchedule)  >  schedule-only / spotSchedule-only
 ### 4. Unresolved references never flap machines
 
 A reference is **Unresolved** when the provider CRD is not installed, the named
-object is absent, it has no `status.active`, or its `Ready` condition is
-`False`/missing. On Unresolved, 5-Spot:
+object is absent, it has no `status.active`, or it carries a `Ready` condition
+whose status is **not `True`**. `Ready` is *recommended, not required*: a
+provider that **omits** `Ready` has its `status.active` taken as authoritative
+(resolved) — only a *present, non-`True`* `Ready` marks the reference
+unresolved, which is how a provider explicitly says "do not trust my `active`
+right now." On Unresolved, 5-Spot:
 
 - sets a `SpotScheduleResolved=False` condition on the `ScheduledMachine` with
   a precise reason (`ProviderCRDNotInstalled`, `ProviderNotFound`,

docs/src/operations/monitoring.md

@@ -69,6 +69,18 @@ All metrics use the `fivespot_` prefix. The full list lives in
 | `fivespot_errors_total` | Counter | `error_type` | Errors by type |
 | `fivespot_finalizer_cleanup_timeouts_total` | Counter | — | Finalizer cleanup timeouts (force-removed; possible orphans) |
 
+#### Spot-schedule providers
+
+Emitted only for `ScheduledMachine`s with a `spec.spotSchedule` reference
+(ADR 0006). Labels are bounded by `namespace` × provider `kind` — never the
+(unbounded) provider or machine name.
+
+| Metric | Type | Labels | Description |
+| ------ | ---- | ------ | ----------- |
+| `fivespot_spot_schedule_resolutions_total` | Counter | `namespace`, `kind`, `result` | Provider resolutions; `result={active\|inactive\|unresolved}` |
+| `fivespot_spot_schedule_resolution_errors_total` | Counter | `namespace`, `kind`, `reason` | Unresolved resolutions by `reason` (`ProviderCRDNotInstalled`, `ProviderNotFound`, `StatusActiveMissing`, `ProviderNotReady`) — the hold-last-state signal to alert on |
+| `fivespot_spot_schedule_transitions_total` | Counter | `namespace`, `kind` | Provider `active`⇄`inactive` transitions; a high rate is the flapping signal |
+
 #### Node drain & eviction
 
 | Metric | Type | Labels | Description |

docs/src/reference/spot-schedule-contract.md

@@ -4,11 +4,15 @@ SPDX-License-Identifier: Apache-2.0
 -->
 # Spot Schedule Provider Contract
 
-> **Status:** Phase 1 — the API types exist (`ScheduledMachine.spec.spotSchedule`
-> on `v1beta1`, the `CapitalMarketsSchedule` reference provider CRD), but the
-> controller-side **resolver and watch** land in later roadmap phases. The
-> contract is **Accepted** in [ADR 0006](https://github.com/finos/5-spot). This
-> page is the authoritative specification a provider author implements against.
+> **Status:** Phase 3 — the controller resolves a `spec.spotSchedule` reference,
+> folds the provider verdict into the machine's active/inactive decision
+> (`status.spotSchedule`), **and now watches referenced providers event-driven**:
+> a provider's `status.active` flip wakes the referencing machines at watch
+> latency (a dynamic per-GVK `watcher` stream, not polling). The
+> reference-provider **controller** that computes
+> `CapitalMarketsSchedule.status.active` lands in Phase 5. The contract is
+> **Accepted** in [ADR 0006](https://github.com/finos/5-spot). This page is the
+> authoritative specification a provider author implements against.
 
 A **spot-schedule provider** is any Kubernetes custom resource in the
 `spotschedules.5spot.finos.org` API group that tells 5-Spot whether a
@@ -57,7 +61,7 @@ spec:
 | `status` field | Required | Type | Meaning |
 |---|:--:|---|---|
 | `active` | **yes** | bool | **The decision.** `true` ⇒ the referencing machine should be up; `false` ⇒ it should be down. |
-| `conditions[type=Ready]` | recommended | condition | Provider health. `Ready=False` (or absent) ⇒ 5-Spot treats the reference as **unresolved**, *not* inactive (see [Unresolved behavior](#unresolved-behavior)). |
+| `conditions[type=Ready]` | recommended | condition | Provider health. A *present* `Ready` whose status is **not `True`** ⇒ 5-Spot treats the reference as **unresolved**, *not* inactive (see [Unresolved behavior](#unresolved-behavior)). An **absent** `Ready` ⇒ `status.active` is taken as authoritative (`Ready` is recommended, not required). |
 | `observedGeneration` | recommended | int64 | The `metadata.generation` the status reflects; lets 5-Spot detect stale status. |
 | `lastTransitionTime` | recommended | RFC 3339 string | When `active` last flipped; used for observability and flap detection. |
 

src/constants.rs

@@ -298,6 +298,19 @@ pub const MAX_RECONCILE_RETRIES: u32 = 10;
 /// steady-state ceiling.
 pub const CHILD_NODE_EVENT_CHANNEL_CAP: usize = 1024;
 
+/// Buffer size for the mpsc channel that carries
+/// `ObjectRef<ScheduledMachine>` from the dynamic spot-schedule provider
+/// watchers into the `Controller::reconcile_on` stream (ADR 0006, Phase 3).
+/// One channel is shared across every per-GVK provider watcher.
+pub const SPOT_SCHEDULE_EVENT_CHANNEL_CAP: usize = 1024;
+
+/// Back-off between retries when a spot-schedule provider watcher cannot yet
+/// resolve its `ApiResource` via discovery (the provider CRD is referenced but
+/// not installed) or its watch stream terminates (e.g. the CRD was deleted).
+/// kube-runtime reconnects transient watch errors internally; this back-off
+/// only governs the discovery re-resolution loop.
+pub const SPOT_SCHEDULE_DISCOVERY_RETRY_SECS: u64 = 30;
+
 /// Maximum number of cached child-cluster `kube::Client` instances kept in
 /// memory by [`crate::reconcilers::child_client::ChildClientCache`].
 ///