Phase 2-4: true event-driven watch, a ScheduledMachine.spec.spotSchedule reference now actually drives the machine's active/inactive decision (pull-on-reconcile) (#83)

* Phase 1: add spotSchedule to CRD and move to v1beta1

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

* Phase 2 and 3: true event-driven watch, a ScheduledMachine.spec.spotSchedule reference now actually drives the machine's active/inactive decision (pull-on-reconcile)

Phase 4: least-privilege RBAC, admission posture, and threat model are now current.

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

---------

Signed-off-by: Erick Bourgeois <erick@jeb.ca>

Author: ebourgeois

.claude/CHANGELOG.md

@@ -9,6 +9,150 @@ The format is based on the regulated environment requirements:
 
 ---
 
+## [2026-06-14 14:30] - Spot-schedule RBAC + admission + threat model (Phase 4)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `deploy/deployment/rbac/clusterrole.yaml`: NEW rule — `get;list;watch` on
+  `spotschedules.5spot.finos.org` `*` (read-only, no write). Unblocks the Phase 3
+  provider watch; least-privilege per ADR 0006 §6 (providers are untrusted
+  inputs the controller never writes).
+- `deploy/admission/validatingadmissionpolicy.yaml`: **security fix** —
+  `matchConstraints.apiVersions` now covers `v1beta1` as well as `v1alpha1`.
+  Since Phase 1 made `v1beta1` the served/storage version, every `v1beta1`
+  ScheduledMachine (i.e. every `spotSchedule` machine) was **bypassing the
+  entire VAP** — the bootstrap/infra allowlist and privilege-escalation guards
+  included. Also: the four schedule rules are now guarded with
+  `!has(object.spec.schedule) || …` (schedule is optional since v1beta1, so a
+  valid spotSchedule-only machine is no longer wrongly rejected), and NEW rules
+  7a (at least one of schedule/spotSchedule) + 7b (spotSchedule group pin)
+  mirror the CRD CEL and `validate_activation_source()`.
+- `docs/src/security/threat-model.md`: NEW actor (spot-schedule provider,
+  untrusted), threat T8a (provider-driven activation control — mitigated:
+  read-only RBAC, same-namespace, group-pinned, auditable) and D5 (provider
+  flapping — partially mitigated: back-off + lifecycle timers + transition
+  metric/alert).
+- `docs/src/security/admission-validation.md`: documented the dual-version
+  match, the optional-schedule guard, and rules 7a/7b.
+- `docs/src/operations/monitoring.md`: suggested `SpotScheduleProviderFlapping`
+  and `SpotScheduleProviderUnresolved` Prometheus alerts.
+- `docs/architecture/calm/architecture.json`: `rel-controller-spot-schedule-watch`
+  least-privilege control evidence now cites the implemented ClusterRole + VAP.
+  `make calm-validate` clean.
+
+### Why
+Phase 4: realize ADR 0006 §6's least-privilege, providers-are-untrusted posture
+in the deployed manifests, and bring the threat model current. The VAP
+version-match fix also closes an admission-bypass regression that Phase 1's
+v1beta1 bump introduced.
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout (re-apply ClusterRole + ValidatingAdmissionPolicy)
+- [ ] Config change only
+- [ ] Documentation only
+
+## [2026-06-14 11:00] - Spot-schedule dynamic event-driven provider watch (Phase 3)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/reconcilers/spot_schedule_watch.rs`: NEW. `ReverseIndex` (pure:
+  `(gvk, ns, name)` → referencing `ScheduledMachine`s, with per-SM record so a
+  changed/removed `spec.spotSchedule` updates precisely) + `provider_key_for`.
+  `SpotScheduleWatchManager` owns the index and lazily runs **one
+  `kube::runtime::watcher` stream per referenced provider GVK** (event-driven,
+  not polling); `sync_watchers` reconciles the running watcher set against the
+  index's referenced GVKs (spawn new, abort orphaned). Per-GVK task resolves the
+  `ApiResource` via `discovery::pinned_kind` with back-off
+  (`SPOT_SCHEDULE_DISCOVERY_RETRY_SECS`, handles CRD-installed-late and
+  CRD-deleted-while-watched), maps each provider event back through the index,
+  and enqueues the referencing SMs.
+- `src/main.rs`: a standalone `ScheduledMachine` reflector feeds SM apply/delete
+  into the manager (index rebuilt from the reflector's initial list on restart —
+  nothing stored outside Kubernetes); a second `Controller::reconcile_on`
+  consumes the manager's mpsc so provider `status.active` flips trigger SM
+  reconciles at watch latency.
+- `src/constants.rs`: `SPOT_SCHEDULE_EVENT_CHANNEL_CAP`,
+  `SPOT_SCHEDULE_DISCOVERY_RETRY_SECS`.
+- `src/reconcilers/mod.rs`: export the watch types + `compose_should_be_active`.
+- Tests: `spot_schedule_watch_tests.rs` (13 — ReverseIndex register/replace/
+  deregister/lookup/gvks, `provider_key_for`, manager observe/forget watcher
+  lifecycle incl. shared-GVK and watcher-stop-on-last-deref);
+  `tests/integration_spot_schedule.rs` (4 — create→active, flip→inactive,
+  delete→unresolved+hold/fail-inactive across a mock kube API, and the watch
+  index→reconcile mapping). 658 lib tests green.
+- Docs: contract page status → Phase 3.
+
+### Why
+Phase 3 of the spot-schedule provider roadmap: remove the Phase-2 pull latency.
+Provider status transitions are now a reconcile **trigger** (a true watch over
+the `spotschedules.5spot.finos.org` group), not something re-checked only on the
+controller's own timer — satisfying the project's watch-not-poll rule. Realizes
+the `rel-controller-spot-schedule-watch` relationship + `flow-spot-schedule-activation`
+already modeled in CALM at Phase 0 (no ADR/CALM change).
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout (new controller image; RBAC for
+      `spotschedules.5spot.finos.org` get/list/watch lands in Phase 4)
+- [ ] Config change only
+- [ ] Documentation only
+
+## [2026-06-13 18:45] - Spot-schedule resolver + composition + status surface (Phase 2)
+
+**Author:** Erick Bourgeois
+
+### Changed
+- `src/reconcilers/spot_schedule.rs`: NEW resolver module. `SpotScheduleVerdict`
+  enum (`Active`/`Inactive`/`Unresolved`); pure `verdict_from_status` (duck-typed
+  `status.active` + `Ready` extraction); async `resolve_spot_schedule`
+  (`discovery::pinned_kind` + `get_opt`). Transient API errors propagate (retry);
+  CRD-not-installed / object-absent / status-missing / not-Ready are `Unresolved`
+  verdicts, never errors or panics. **Ready semantics finalized:** absent `Ready`
+  ⇒ `active` authoritative; present non-`True` `Ready` ⇒ unresolved (resolves an
+  internal inconsistency in ADR 0006 §4 / contract table — both updated).
+- `src/reconcilers/helpers.rs`: NEW `compose_should_be_active` — AND composition
+  of the inline schedule and the provider verdict, with hold-last-state on
+  Unresolved (fail-inactive when never resolved). `killSwitch` precedence is
+  upstream.
+- `src/crd.rs`: replaced Phase-1 `effective_schedule()` with `is_enabled()` (a
+  spotSchedule-only machine is always enabled; the provider governs activity).
+  NEW `SpotScheduleStatus` + `ScheduledMachineStatus.spot_schedule` field (the
+  durable hold-last-state surface: `resolved`/`active`/`reason`/`message`/
+  `providerGeneration`/`lastTransitionTime`).
+- `src/reconcilers/scheduled_machine.rs`: `reconcile_inner` now evaluates the
+  optional inline schedule, resolves the provider (pull-on-reconcile), composes
+  `should_be_active`, records metrics, and patches `status.spotSchedule`
+  (`persist_spot_schedule_status`, a `/status` merge patch that coexists with the
+  phase/condition writes). Handler `enabled` guards use `is_enabled()`.
+- `src/metrics.rs`: NEW `fivespot_spot_schedule_resolutions_total{namespace,kind,result}`,
+  `_resolution_errors_total{namespace,kind,reason}`,
+  `_transitions_total{namespace,kind}` (+ record fns; bounded cardinality, never
+  keyed by name).
+- `deploy/crds/scheduledmachine.yaml` regenerated (status.spotSchedule);
+  `docs/src/reference/api.md` regenerated.
+- Tests: `spot_schedule_tests.rs` (verdict truth table + 3 mock-client resolve
+  paths), `helpers_tests.rs` (12-case composition truth table),
+  `scheduled_machine_tests.rs` (persist `/status` patch), `metrics_tests.rs`,
+  `crd_tests.rs` (status round-trip + is_enabled). 645 lib tests green.
+- Docs: ADR 0006 + contract page `Ready` clarification; contract page status →
+  Phase 2; `docs/src/operations/monitoring.md` spot-schedule metrics table.
+
+### Why
+Phase 2 of the spot-schedule provider roadmap: make a `spec.spotSchedule`
+reference actually drive the machine's active/inactive decision (pull-on-reconcile),
+with the ADR-specified hold-last-state fail-safe and an observable status surface.
+The event-driven watch (Phase 3) and the reference-provider controller (Phase 5)
+build on this.
+
+### Impact
+- [ ] Breaking change
+- [x] Requires cluster rollout (apply regenerated CRDs + new controller image)
+- [ ] Config change only
+- [ ] Documentation only
+
 ## [2026-06-13 16:30] - Spot-schedule provider API: CRD scaffolding + ScheduledMachine v1beta1 (Phase 1)
 
 **Author:** Erick Bourgeois

deploy/admission/validatingadmissionpolicy.yaml

@@ -27,8 +27,12 @@ spec:
 
   matchConstraints:
     resourceRules:
+      # Both served versions must be validated (ADR 0007). v1beta1 is the
+      # storage/current version (adds spec.spotSchedule and an optional
+      # spec.schedule); v1alpha1 is the frozen, deprecated legacy version. A
+      # version omitted here would BYPASS every validation below.
       - apiGroups: ["5spot.finos.org"]
-        apiVersions: ["v1alpha1"]
+        apiVersions: ["v1alpha1", "v1beta1"]
         resources: ["scheduledmachines"]
         operations: ["CREATE", "UPDATE"]
 
@@ -117,11 +121,19 @@ spec:
         (format: <positive integer> followed by s, m, or h)
       reason: Invalid
 
+    # NOTE on the schedule rules below: `spec.schedule` is OPTIONAL since
+    # v1beta1 (ADR 0006) — a machine may instead delegate to a spotSchedule
+    # provider. Each schedule rule is therefore guarded with
+    # `!has(object.spec.schedule) || …` so a valid spotSchedule-only machine is
+    # not rejected. The "at least one of schedule/spotSchedule" invariant is
+    # enforced by rule 7a below (and by the CRD's own x-kubernetes-validations).
+
     # ── 4. Cron XOR explicit day/hour windows ────────────────────────────────
     # Using cron together with daysOfWeek or hoursOfDay is ambiguous.
     # The controller silently ignores daysOfWeek/hoursOfDay when cron is set;
     # we reject the spec at admission to prevent silent data loss.
     - expression: |
+        !has(object.spec.schedule) ||
         !has(object.spec.schedule.cron) ||
         (object.spec.schedule.daysOfWeek.size() == 0 &&
          object.spec.schedule.hoursOfDay.size() == 0)
@@ -132,6 +144,7 @@ spec:
 
     # ── 5. Without cron, both day and hour windows must be provided ───────────
     - expression: |
+        !has(object.spec.schedule) ||
         has(object.spec.schedule.cron) ||
         (object.spec.schedule.daysOfWeek.size() > 0 &&
          object.spec.schedule.hoursOfDay.size() > 0)
@@ -145,6 +158,7 @@ spec:
     # or a comma-separated combination (mon-wed,fri-sun).
     # Trivially true for empty arrays (cron path).
     - expression: |
+        !has(object.spec.schedule) ||
         object.spec.schedule.daysOfWeek.all(d,
           d.matches('^(mon|tue|wed|thu|fri|sat|sun)(-(mon|tue|wed|thu|fri|sat|sun))?(,(mon|tue|wed|thu|fri|sat|sun)(-(mon|tue|wed|thu|fri|sat|sun))?)*$'))
       message: >-
@@ -158,13 +172,35 @@ spec:
     # at runtime by the schedule evaluator.
     # Trivially true for empty arrays (cron path).
     - expression: |
+        !has(object.spec.schedule) ||
         object.spec.schedule.hoursOfDay.all(h,
           h.matches('^[0-9]{1,2}(-[0-9]{1,2})?(,[0-9]{1,2}(-[0-9]{1,2})?)*$'))
       message: >-
         spec.schedule.hoursOfDay items must be hours or ranges
         (e.g. '9', '9-17', '0-9,18-23')
       reason: Invalid
 
+    # ── 7a. At least one activation source ────────────────────────────────────
+    # A machine must declare an inline schedule, a spotSchedule provider, or
+    # both (ADR 0006). Mirrors the CRD x-kubernetes-validations and the runtime
+    # validate_activation_source() in src/reconcilers/helpers.rs.
+    - expression: "has(object.spec.schedule) || has(object.spec.spotSchedule)"
+      message: >-
+        at least one of spec.schedule or spec.spotSchedule must be set
+      reason: Invalid
+
+    # ── 7b. spotSchedule.apiVersion group pin ─────────────────────────────────
+    # The provider reference group MUST be spotschedules.5spot.finos.org
+    # (ADR 0006 §2 / §6) — providers are untrusted inputs and the controller's
+    # read-only RBAC is scoped to exactly this group. Mirrors the CRD field CEL
+    # and the runtime group check.
+    - expression: |
+        !has(object.spec.spotSchedule) ||
+        object.spec.spotSchedule.apiVersion.startsWith('spotschedules.5spot.finos.org/')
+      message: >-
+        spec.spotSchedule.apiVersion group must be spotschedules.5spot.finos.org
+      reason: Invalid
+
     # ── 8. bootstrapSpec.apiVersion must use a namespaced API group ───────────
     # Core API versions (e.g. "v1") have no '/' and must never be used for
     # bootstrap resources — rejects the same case as validate_api_group() at runtime.

deploy/crds/scheduledmachine.yaml

@@ -673,6 +673,47 @@ spec:
                   (Pending, ShuttingDown, Inactive, Disabled, Terminated, Error) is
                   reported as `False`.
                 type: boolean
+              spotSchedule:
+                description: |-
+                  Spot-schedule provider resolution state (ADR 0006), present only when
+                  `spec.spotSchedule` is set. Carries the last resolved/held provider
+                  `active` value (the input to hold-last-state composition), the
+                  resolution reason/message, and the provider's observed generation.
+                nullable: true
+                properties:
+                  active:
+                    description: |-
+                      Last known provider `status.active`. Held across an unresolved reconcile
+                      (hold-last-state); `None` until the provider first resolves.
+                    nullable: true
+                    type: boolean
+                  lastTransitionTime:
+                    description: When `active` last transitioned (RFC3339).
+                    nullable: true
+                    type: string
+                  message:
+                    description: Human-readable resolution detail.
+                    nullable: true
+                    type: string
+                  providerGeneration:
+                    description: The provider's `status.observedGeneration` at the last resolution.
+                    format: int64
+                    nullable: true
+                    type: integer
+                  reason:
+                    description: Machine-readable resolution reason (a `REASON_SPOT_SCHEDULE_*` value).
+                    nullable: true
+                    type: string
+                  resolved:
+                    description: |-
+                      Whether the referenced provider resolved into an authoritative verdict
+                      this reconcile. `false` mirrors a `SpotScheduleResolved=False`
+                      condition — the provider CRD is absent, the object is missing, it
+                      exposes no `status.active`, or it is not `Ready`.
+                    type: boolean
+                required:
+                - resolved
+                type: object
             type: object
         required:
         - spec
@@ -1204,6 +1245,47 @@ spec:
                   (Pending, ShuttingDown, Inactive, Disabled, Terminated, Error) is
                   reported as `False`.
                 type: boolean
+              spotSchedule:
+                description: |-
+                  Spot-schedule provider resolution state (ADR 0006), present only when
+                  `spec.spotSchedule` is set. Carries the last resolved/held provider
+                  `active` value (the input to hold-last-state composition), the
+                  resolution reason/message, and the provider's observed generation.
+                nullable: true
+                properties:
+                  active:
+                    description: |-
+                      Last known provider `status.active`. Held across an unresolved reconcile
+                      (hold-last-state); `None` until the provider first resolves.
+                    nullable: true
+                    type: boolean
+                  lastTransitionTime:
+                    description: When `active` last transitioned (RFC3339).
+                    nullable: true
+                    type: string
+                  message:
+                    description: Human-readable resolution detail.
+                    nullable: true
+                    type: string
+                  providerGeneration:
+                    description: The provider's `status.observedGeneration` at the last resolution.
+                    format: int64
+                    nullable: true
+                    type: integer
+                  reason:
+                    description: Machine-readable resolution reason (a `REASON_SPOT_SCHEDULE_*` value).
+                    nullable: true
+                    type: string
+                  resolved:
+                    description: |-
+                      Whether the referenced provider resolved into an authoritative verdict
+                      this reconcile. `false` mirrors a `SpotScheduleResolved=False`
+                      condition — the provider CRD is absent, the object is missing, it
+                      exposes no `status.active`, or it is not `Ready`.
+                    type: boolean
+                required:
+                - resolved
+                type: object
             type: object
         required:
         - spec

deploy/deployment/rbac/clusterrole.yaml

@@ -28,6 +28,20 @@ rules:
     resources: ["scheduledmachines/finalizers"]
     verbs: ["update"]
 
+  # Spot-schedule provider resources (ADR 0006). The controller WATCHES and
+  # READS provider objects in the spotschedules.5spot.finos.org group to resolve
+  # `spec.spotSchedule` into a duck-typed status.active verdict — it never reads
+  # the provider spec semantics and NEVER writes a provider object. Resources is
+  # a wildcard because 5-Spot is provider-agnostic (CapitalMarketsSchedule today,
+  # any third-party provider kind tomorrow); verbs are deliberately read-only —
+  # providers are untrusted inputs whose blast radius is bounded to the
+  # same-namespace ScheduledMachines that reference them (ADR 0006 §6). The
+  # provider's OWN controller holds update/patch on its kind's /status (a
+  # separate Role it ships, not this ClusterRole).
+  - apiGroups: ["spotschedules.5spot.finos.org"]
+    resources: ["*"]
+    verbs: ["get", "list", "watch"]
+
   # CAPI Machine management
   - apiGroups: ["cluster.x-k8s.io"]
     resources: ["machines", "machines/status"]

docs/adr/0006-pluggable-spot-schedule-provider-contract.md

@@ -109,8 +109,12 @@ killSwitch  >  (schedule AND spotSchedule)  >  schedule-only / spotSchedule-only
 ### 4. Unresolved references never flap machines
 
 A reference is **Unresolved** when the provider CRD is not installed, the named
-object is absent, it has no `status.active`, or its `Ready` condition is
-`False`/missing. On Unresolved, 5-Spot:
+object is absent, it has no `status.active`, or it carries a `Ready` condition
+whose status is **not `True`**. `Ready` is *recommended, not required*: a
+provider that **omits** `Ready` has its `status.active` taken as authoritative
+(resolved) — only a *present, non-`True`* `Ready` marks the reference
+unresolved, which is how a provider explicitly says "do not trust my `active`
+right now." On Unresolved, 5-Spot:
 
 - sets a `SpotScheduleResolved=False` condition on the `ScheduledMachine` with
   a precise reason (`ProviderCRDNotInstalled`, `ProviderNotFound`,

docs/architecture/calm/architecture.json

@@ -724,7 +724,7 @@
               "config": {
                 "framework": "NIST SP 800-53 Rev. 5",
                 "controls": ["AC-3", "AC-4", "AC-6", "SI-10"],
-                "evidence": "docs/adr/0006-pluggable-spot-schedule-provider-contract.md, docs/src/security/threat-model.md"
+                "evidence": "deploy/deployment/rbac/clusterrole.yaml (spotschedules.5spot.finos.org get/list/watch only), deploy/admission/validatingadmissionpolicy.yaml (rules 7a/7b), docs/adr/0006-pluggable-spot-schedule-provider-contract.md, docs/src/security/threat-model.md"
               }
             }
           ]