Skip to content

Move to using providers instead of inline time based schdules, this will solidify the provider contract as a first class API #87

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ebourgeois merged 1 commit into from
Jun 17, 2026
Merged

Move to using providers instead of inline time based schdules, this will solidify the provider contract as a first class API #87

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
149 changes: 149 additions & 0 deletions .claude/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,155 @@ The format is based on the regulated environment requirements:

---

## [2026-06-17 09:00] - Pin deploy manifest image tags to the release version; crdgen back to stdout

**Author:** Erick Bourgeois

### Changed
- `Makefile`: NEW `set-image-version VERSION=…` target — rewrites every
`ghcr.io/finos/5-spot*` image tag under `deploy/` (controller, both
spot-schedule providers, reclaim agent, kata-config agent) to the given
version. Tag-suffix-only regex, so it is independent of the tag committed in
git. Added to `.PHONY`.
- `.github/workflows/build.yaml` (`package-deploy-manifests`, release-only):
added a "Pin deployment image versions" step running
`make set-image-version VERSION=${{ needs.extract-version.outputs.tag_name }}`
before tarring, so the released `deploy-manifests.tar.gz` references the image
variant just built/pushed. Its "Regenerate CRDs" step now runs `make crds`.
- `src/bin/crdgen.rs`: reverted to **stdout** output (the binary no longer writes
files or assumes `deploy/crds/` exists — matches `crddoc`). Args are parsed via
`clap` (a `ValueEnum` selector), consistent with the other binaries and
avoiding raw `std::env::args` (resolves Semgrep `rust.lang.security.args.args`,
code-scanning alert #175). No-arg prints all CRDs as a `---`-separated
multi-doc stream (pipe-to-`kubectl`-friendly); a selector (`scheduledmachine` /
`timebasedspotschedule` / `capitalmarketsschedule`) prints just one. The
file-writing approach was only introduced in the Phase-1 multi-version commit to
host the schedule/spotSchedule CEL injection, which ADR 0009 removed — so crdgen
is pure serialization again.
- `Makefile` `crds` target: now owns the file layout, redirecting each selector
to `deploy/crds/<selector>.yaml`.
- `deploy/crds/capitalmarketsschedule.yaml`: regenerated (picks up the
`TimeBasedSpotScheduleSpec` doc-comment fix in its description).

### Why
The deployment manifests hardcoded `:v0.1.0`; the release pipeline built/pushed
`ghcr.io/finos/5-spot:<tag>` but shipped manifests still pointed at the stale
tag. Pinning at release time keeps the shipped manifests in lockstep with the
image that was actually built. crdgen returning to stdout keeps path/layout
decisions with the caller (Makefile/CI), not baked into the binary.

### Impact
- [ ] Breaking change
- [x] Requires cluster rollout
- [ ] Config change only
- [ ] Documentation only

---

## [2026-06-16 13:00] - ADR 0009: examples, time-based provider deploy manifests, and prose docs

**Author:** Erick Bourgeois

### Changed
- `src/crd.rs`: `ScheduledMachineSpec.schedule` is now a **required**
`SpotScheduleRef` (was `Option<ScheduleSpec>`); removed `spec.spotSchedule`;
added top-level `spec.enabled` master switch; removed `ScheduleSpec` and the
`v1alpha1` `ScheduledMachine` module; added the `TimeBasedSpotSchedule` CRD
(spec + status) in `spotschedules.5spot.finos.org`. `is_enabled()` now reads
`spec.enabled`.
- `src/providers/time_based.rs` (+tests) and `src/bin/spot_schedule_time_based.rs`:
NEW core/default provider controller — `is_active_at`/`next_transition`/
`reconcile`, mirroring `capital_markets`; publishes `status.active`.
- `src/reconcilers/{scheduled_machine,helpers,spot_schedule_watch}.rs`: drive
activation solely from the single `spec.schedule` provider verdict; removed
`evaluate_schedule`; simplified `compose_should_be_active` (verdict + hold-last
-state); replaced `validate_activation_source` with `validate_schedule_ref`;
emergency-reclaim loop-breaker now patches `spec.enabled=false` (never the
shared provider object); watch reverse-index keys off `spec.schedule`.
- `src/metrics.rs` (+tests): added `fivespot_time_based_active` /
`fivespot_time_based_transitions_total` + record fns.
- `src/bin/{crdgen,crddoc}.rs`: emit the `TimeBasedSpotSchedule` CRD; single
`v1beta1` `ScheduledMachine` (dropped the multi-version merge + schedule/
spotSchedule XOR CEL); api.md regenerated.
- `deploy/admission/validatingadmissionpolicy.yaml`: removed the inline-schedule
rules (cron/day/hour) and the "at least one" rule; the remaining schedule rule
pins `spec.schedule.apiVersion` to the provider group; `v1beta1`-only.
- `docs/adr/0009-*.md` (+ index; amended 0006 §1/§3 and 0007);
`docs/architecture/calm/architecture.json` (provider model + flows; validated,
diagrams regenerated).
- `examples/scheduledmachine-*.yaml`: migrated all to `5spot.finos.org/v1beta1`,
added `spec.enabled: true`, and replaced every inline `schedule:` / removed
`spotSchedule:` block with a single `spec.schedule:` provider reference. Basic
/ weekend / tainted / bad-taint / kata / child-cluster now reference a
`TimeBasedSpotSchedule`; `scheduledmachine-spot-schedule.yaml` references
`CapitalMarketsSchedule` (`nyse-equities`) alone (no more AND-composition).
- `examples/timebasedspotschedule.yaml`: NEW companion file defining the
`business-hours` / `weekend-only` (ns `default`) and `business-hours-toronto`
(ns `hosted-cluster-alpha`) provider objects the SM examples reference.
- `examples/capitalmarketsschedule.yaml`, `examples/workshop/*`: updated
comments / shape to `spec.schedule` reference + `spec.enabled`; workshop now
installs the time-based provider and toggles `spec.enabled` / `tbss`.
- `deploy/spot-schedule-providers/time-based/`: NEW provider deploy bundle
(serviceaccount / clusterrole / clusterrolebinding / deployment /
kustomization), mirroring the capital-markets bundle — least-privilege RBAC
(get/list/watch `timebasedspotschedules`, update/patch its `/status`),
hardened `Deployment` running `spot-schedule-time-based`.
- `docs/src/**` and `docs/mkdocs.yml`: rewrote all spot-schedule prose to the
ADR 0009 shape (required `spec.schedule` provider ref; `TimeBasedSpotSchedule`
as the default first-party provider; `spec.enabled` master switch; emergency
reclaim sets `spec.enabled`; VAP single group-pin rule 4 / `v1beta1`-only
matchConstraints). Added a new `docs/src/guides/time-based-schedule.md` guide
and listed it in the nav.

### Why
ADR 0009 reified the former inline `ScheduledMachine.spec.schedule` window into
the first-party `TimeBasedSpotSchedule` provider, made `spec.schedule` a required
provider reference, dropped the SM `v1alpha1` version and the `spotSchedule`
field, and introduced `spec.enabled`. Examples, deploy manifests, and docs had
to match the new contract.

### Impact
- [x] Breaking change (pre-release CRD contract: `spec.schedule` is now a
required provider reference, `spec.spotSchedule` removed, SM `v1alpha1` dropped)
- [x] Requires cluster rollout (new CRD + provider controller + RBAC; re-apply
`deploy/crds/` and the time-based provider bundle)
- [ ] Config change only
- [ ] Documentation only

---

**Author:** Erick Bourgeois

### Changed
- `docs/src/guides/create-your-own-provider.md`: NEW standalone guide — a
complete, copy-pasteable walkthrough building a minimal `ManualSchedule`
provider end to end: CRD (namespaced, `status` subresource), controller
(status-patch reconcile, `Ready` semantics, event-driven not polling),
least-privilege RBAC, deployment, referencing from a `ScheduledMachine`,
verification, and additive versioning. Language-agnostic.
- `docs/mkdocs.yml`: grouped both provider guides under a **Spot Schedule
Providers** sub-section in Guides (CapitalMarketsSchedule + Create Your Own
Provider) — makes the existing capital-markets guide more discoverable.
- `docs/src/reference/spot-schedule-contract.md`: replaced the inline
"implement your own" walkthrough with a pointer to the new guide (reference =
spec, guide = how-to); added both guides to See-also.
- `docs/src/guides/capital-markets-schedule.md`: added a See-also section
cross-linking the concept, contract, and the new provider guide.

### Why
The "create your own provider" how-to was buried inside the contract *reference*
page; operators asked for it (and the CapitalMarketsSchedule guide) as
first-class entries under **Guides**. Splitting how-to from spec also keeps the
reference page focused.

### Impact
- [ ] Breaking change
- [ ] Requires cluster rollout
- [ ] Config change only
- [x] Documentation only

---

## [2026-06-15 10:30] - Restore dropped spot-schedule metric functions (build fix)

**Author:** Erick Bourgeois
Expand Down
11 changes: 10 additions & 1 deletion .github/workflows/build.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1288,8 +1288,17 @@ jobs:
- name: Cache cargo dependencies
uses: firestoned/github-actions/rust/cache-cargo@53b483254bc648903c364ee3c73a546d0936a91e # v1.3.6

# `make crds` writes one CRD per file under deploy/crds/ (crdgen itself
# only prints to stdout — the Makefile owns the file layout).
- name: Regenerate CRDs
run: cargo run --quiet --bin crdgen > deploy/crds/scheduledmachine.yaml
run: make crds

# Pin every deploy/ image tag to the release tag so the packaged manifests
# reference the image variant just built and pushed (ghcr.io/finos/5-spot
# and the 5-spot-* agent images all share this tag). See `make
# set-image-version`.
- name: Pin deployment image versions to the release tag
run: make set-image-version VERSION=${{ needs.extract-version.outputs.tag_name }}

- name: Package deploy manifests
run: tar czf deploy-manifests.tar.gz deploy/
Expand Down
30 changes: 27 additions & 3 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Copyright (c) 2025 Erick Bourgeois, RBC Capital Markets
# SPDX-License-Identifier: Apache-2.0

.PHONY: help install build build-debug build-linux-amd64 build-linux-arm64 build-macos-arm64 prepare-binaries-linux-amd64 prepare-binaries-linux-arm64 test test-lib lint format clean crds crddoc docs docs-serve docs-clean docs-rustdoc calm-diagrams calm-validate run-local docker-build docker-build-amd64 docker-build-arm64 docker-build-chainguard docker-push docker-buildx docker-buildx-chainguard gitleaks gitleaks-install install-git-hooks security-scan-local sbom audit vexctl-install vex-validate vex-assemble vex-auto-presence vex-auto-reachability vex-auto vex-auto-check kind-install kind-create kind-delete kind-load kind-deploy kind-example kind-setup kind-status
.PHONY: help install build build-debug build-linux-amd64 build-linux-arm64 build-macos-arm64 prepare-binaries-linux-amd64 prepare-binaries-linux-arm64 test test-lib lint format clean crds crddoc docs docs-serve docs-clean docs-rustdoc calm-diagrams calm-validate run-local docker-build docker-build-amd64 docker-build-arm64 docker-build-chainguard docker-push docker-buildx docker-buildx-chainguard gitleaks gitleaks-install install-git-hooks security-scan-local sbom audit vexctl-install vex-validate vex-assemble vex-auto-presence vex-auto-reachability vex-auto vex-auto-check set-image-version kind-install kind-create kind-delete kind-load kind-deploy kind-example kind-setup kind-status

# CALM (FINOS Common Architecture Language Model) configuration
CALM_CLI_VERSION ?= 1.37.0
Expand Down Expand Up @@ -172,16 +172,40 @@ run-local: ## Run operator locally
# Code Generation
# ============================================================

# crdgen prints one CRD to stdout per selector; this target owns the file layout
# (the binary makes no path assumptions). Keep this list in sync with crdgen's
# SELECTORS and the committed deploy/crds/*.yaml filenames.
CRD_SELECTORS ?= scheduledmachine timebasedspotschedule capitalmarketsschedule

crds: ## Generate CRD YAML files from Rust types
@echo "Generating CRD YAML files from src/crd.rs..."
@cargo run --quiet --bin crdgen
@echo "✓ CRD YAML files generated under deploy/crds/"
@for sel in $(CRD_SELECTORS); do \
cargo run --quiet --bin crdgen -- "$$sel" > "deploy/crds/$$sel.yaml"; \
echo "✓ wrote deploy/crds/$$sel.yaml"; \
done

crddoc: ## Generate API documentation from CRD types
@echo "Generating API documentation..."
@cargo run --quiet --bin crddoc > docs/src/reference/api.md
@echo "✓ API documentation generated: docs/src/reference/api.md"

# Pin every 5-Spot image tag under deploy/ to VERSION so the shipped manifests
# match the image actually built/pushed at release. The controller and both
# spot-schedule providers share the main `ghcr.io/finos/5-spot` image; the
# reclaim- and kata-config agents use their own `ghcr.io/finos/5-spot-*` repos —
# all carry the same release tag. The regex replaces only the tag suffix, so it
# is independent of whatever tag is committed in git. Used by the release
# `package-deploy-manifests` CI job; safe to run locally to preview the diff.
set-image-version: ## Pin deploy/ image tags to VERSION (e.g. make set-image-version VERSION=v0.2.0)
@test -n "$(VERSION)" || { \
echo "ERROR: VERSION is required, e.g. make set-image-version VERSION=v0.2.0"; exit 1; }
@files=$$(grep -rl "image:[[:space:]]*ghcr.io/finos/5-spot" deploy/); \
if [ -z "$$files" ]; then echo "ERROR: no 5-Spot image references found under deploy/"; exit 1; fi; \
for f in $$files; do \
perl -i -pe 's{(image:\s*ghcr\.io/finos/5-spot[\w./-]*):\S+}{$${1}:$(VERSION)}g' "$$f"; \
echo "✓ pinned 5-Spot images in $$f to $(VERSION)"; \
done

# ============================================================
# Documentation
# ============================================================
Expand Down
96 changes: 17 additions & 79 deletions deploy/admission/validatingadmissionpolicy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,12 +27,11 @@ spec:

matchConstraints:
resourceRules:
# Both served versions must be validated (ADR 0007). v1beta1 is the
# storage/current version (adds spec.spotSchedule and an optional
# spec.schedule); v1alpha1 is the frozen, deprecated legacy version. A
# version omitted here would BYPASS every validation below.
# Single served/storage version since ADR 0009 (the pre-release v1alpha1
# ScheduledMachine was dropped). spec.schedule is a required provider
# reference. A version omitted here would BYPASS every validation below.
- apiGroups: ["5spot.finos.org"]
apiVersions: ["v1alpha1", "v1beta1"]
apiVersions: ["v1beta1"]
resources: ["scheduledmachines"]
operations: ["CREATE", "UPDATE"]

Expand Down Expand Up @@ -121,84 +120,23 @@ spec:
(format: <positive integer> followed by s, m, or h)
reason: Invalid

# NOTE on the schedule rules below: `spec.schedule` is OPTIONAL since
# v1beta1 (ADR 0006) — a machine may instead delegate to a spotSchedule
# provider. Each schedule rule is therefore guarded with
# `!has(object.spec.schedule) || …` so a valid spotSchedule-only machine is
# not rejected. The "at least one of schedule/spotSchedule" invariant is
# enforced by rule 7a below (and by the CRD's own x-kubernetes-validations).
# NOTE on the schedule rule below: since ADR 0009, `spec.schedule` is a
# REQUIRED reference to a spot-schedule provider object (apiVersion/kind/
# name in the spotschedules.5spot.finos.org group) — the former inline time
# window moved to the TimeBasedSpotSchedule provider CRD, which carries its
# own day/hour/timezone schema validation. The only ScheduledMachine-side
# invariant left for this VAP is the provider group pin (rule 4). The
# "required" and structural shape are enforced by the CRD schema itself.

# ── 4. Cron XOR explicit day/hour windows ────────────────────────────────
# Using cron together with daysOfWeek or hoursOfDay is ambiguous.
# The controller silently ignores daysOfWeek/hoursOfDay when cron is set;
# we reject the spec at admission to prevent silent data loss.
- expression: |
!has(object.spec.schedule) ||
!has(object.spec.schedule.cron) ||
(object.spec.schedule.daysOfWeek.size() == 0 &&
object.spec.schedule.hoursOfDay.size() == 0)
message: >-
spec.schedule: cron is mutually exclusive with daysOfWeek and hoursOfDay —
set one or the other, not both
reason: Invalid

# ── 5. Without cron, both day and hour windows must be provided ───────────
- expression: |
!has(object.spec.schedule) ||
has(object.spec.schedule.cron) ||
(object.spec.schedule.daysOfWeek.size() > 0 &&
object.spec.schedule.hoursOfDay.size() > 0)
message: >-
spec.schedule: when cron is not set, both daysOfWeek and hoursOfDay must be
non-empty to define an active window
reason: Invalid

# ── 6. daysOfWeek item format ─────────────────────────────────────────────
# Each element must be a day name (mon…sun), an inclusive range (mon-fri),
# or a comma-separated combination (mon-wed,fri-sun).
# Trivially true for empty arrays (cron path).
- expression: |
!has(object.spec.schedule) ||
object.spec.schedule.daysOfWeek.all(d,
d.matches('^(mon|tue|wed|thu|fri|sat|sun)(-(mon|tue|wed|thu|fri|sat|sun))?(,(mon|tue|wed|thu|fri|sat|sun)(-(mon|tue|wed|thu|fri|sat|sun))?)*$'))
message: >-
spec.schedule.daysOfWeek items must be day names or ranges
(e.g. 'mon', 'mon-fri', 'mon-wed,fri-sun')
reason: Invalid

# ── 7. hoursOfDay item format ─────────────────────────────────────────────
# Each element must be a 1-2 digit hour, a range (9-17), or a comma-
# separated combination (0-9,18-23). Numeric bounds (0-23) are enforced
# at runtime by the schedule evaluator.
# Trivially true for empty arrays (cron path).
- expression: |
!has(object.spec.schedule) ||
object.spec.schedule.hoursOfDay.all(h,
h.matches('^[0-9]{1,2}(-[0-9]{1,2})?(,[0-9]{1,2}(-[0-9]{1,2})?)*$'))
message: >-
spec.schedule.hoursOfDay items must be hours or ranges
(e.g. '9', '9-17', '0-9,18-23')
reason: Invalid

# ── 7a. At least one activation source ────────────────────────────────────
# A machine must declare an inline schedule, a spotSchedule provider, or
# both (ADR 0006). Mirrors the CRD x-kubernetes-validations and the runtime
# validate_activation_source() in src/reconcilers/helpers.rs.
- expression: "has(object.spec.schedule) || has(object.spec.spotSchedule)"
message: >-
at least one of spec.schedule or spec.spotSchedule must be set
reason: Invalid

# ── 7b. spotSchedule.apiVersion group pin ─────────────────────────────────
# ── 4. schedule.apiVersion group pin ──────────────────────────────────────
# The provider reference group MUST be spotschedules.5spot.finos.org
# (ADR 0006 §2 / §6) — providers are untrusted inputs and the controller's
# read-only RBAC is scoped to exactly this group. Mirrors the CRD field CEL
# and the runtime group check.
# (ADR 0006 §2 / §6, ADR 0009) — providers are untrusted inputs and the
# controller's read-only RBAC is scoped to exactly this group. Mirrors the
# CRD field CEL and the runtime validate_schedule_ref() check.
- expression: |
!has(object.spec.spotSchedule) ||
object.spec.spotSchedule.apiVersion.startsWith('spotschedules.5spot.finos.org/')
object.spec.schedule.apiVersion.startsWith('spotschedules.5spot.finos.org/')
message: >-
spec.spotSchedule.apiVersion group must be spotschedules.5spot.finos.org
spec.schedule.apiVersion group must be spotschedules.5spot.finos.org
reason: Invalid

# ── 8. bootstrapSpec.apiVersion must use a namespaced API group ───────────
Expand Down
Loading
Loading