From 987ce99e1236dc47fa5ad7371983b681a4782940 Mon Sep 17 00:00:00 2001 From: Erick Bourgeois Date: Sun, 31 May 2026 08:23:32 -0400 Subject: [PATCH] Initial, simple workshop Signed-off-by: Erick Bourgeois --- .claude/CHANGELOG.md | 31 ++ .claude/CLAUDE.md | 20 ++ .claude/SKILL.md | 6 + .../rules/architecture-driven-development.md | 100 ++++++ ...1-adopt-architecture-driven-development.md | 85 ++++++ docs/adr/README.md | 39 +++ docs/adr/template.md | 30 ++ examples/workshop/README.md | 288 ++++++++++++++++++ examples/workshop/kind-management.yaml | 20 ++ .../scheduledmachine-business-hours.yaml | 74 +++++ examples/workshop/teardown.sh | 25 ++ examples/workshop/workload-cluster.yaml | 92 ++++++ 12 files changed, 810 insertions(+) create mode 100644 .claude/rules/architecture-driven-development.md create mode 100644 docs/adr/0001-adopt-architecture-driven-development.md create mode 100644 docs/adr/README.md create mode 100644 docs/adr/template.md create mode 100644 examples/workshop/README.md create mode 100644 examples/workshop/kind-management.yaml create mode 100644 examples/workshop/scheduledmachine-business-hours.yaml create mode 100755 examples/workshop/teardown.sh create mode 100644 examples/workshop/workload-cluster.yaml diff --git a/.claude/CHANGELOG.md b/.claude/CHANGELOG.md index 3624b20..ba4151c 100644 --- a/.claude/CHANGELOG.md +++ b/.claude/CHANGELOG.md @@ -9,6 +9,37 @@ The format is based on the regulated environment requirements: --- +## [2026-05-31 00:00] - Adopt Architecture Driven Development (ADD) methodology + +**Author:** Erick Bourgeois + +### Changed +- `docs/adr/`: new directory for Architecture Decision Records — + `template.md`, `README.md` (conventions + index), and + `0001-adopt-architecture-driven-development.md` (Accepted) recording the + adoption decision and rationale. +- `.claude/rules/architecture-driven-development.md`: new rule file defining the + ADD methodology (`ADR → CALM → TDD → implement → docs`), adapted to 5-Spot + (`src/crd.rs` source of truth, `make crds`/`crddoc`, CALM targets, + `cargo-quality`/`sync-docs`, roadmaps external at `~/dev/roadmaps/`). +- `.claude/CLAUDE.md`: added ADD as the governing methodology (new section + as + the first "CRITICAL Coding Pattern"); links to the rule file and ADR-0001. +- `.claude/SKILL.md`: added an ADD gate to the top of `pre-commit-checklist` + (ADR written/indexed; CALM updated or "no CALM impact"; ADR precedes code). + +### Why +5-Spot already had strong TDD and a CALM model, but no first-class, ordered way +to record *why* an architecturally significant change was made before building +it. ADD (per the sibling banlieue project) layers ADR + CALM on top of TDD so +decisions are auditable and traceable — fitting the regulated context — instead +of reconstructed from diffs and changelog entries. See ADR-0001. + +### Impact +- [ ] Breaking change +- [ ] Requires cluster rollout +- [ ] Config change only +- [x] Process / documentation only + ## [2026-05-30 12:00] - Revert docs deploy trigger to release:published (CodeQL: pwn-request + cache poisoning) **Author:** Erick Bourgeois diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index e4bd16a..26894be 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -6,6 +6,7 @@ > **Project Focus**: Time-based scheduling of physical machines in k0smotron clusters. This controller manages machine lifecycle (add/remove from cluster) based on configurable time schedules with timezone support. > > **CRITICAL Coding Patterns**: +> - **Architecture Driven Development (ADD)**: For any architecturally significant change, architecture is decided, recorded, and visualized BEFORE code: `ADR → CALM → TDD → implement → docs`, in that order. ADRs (`docs/adr/`) and CALM diagrams are first-class deliverables, equal to code and tests. Full rule: `.claude/rules/architecture-driven-development.md`. ADD wraps TDD below — it does not replace it. > - **Test-Driven Development (TDD)**: ALWAYS write tests FIRST before implementing functionality. Write failing tests that define the expected behavior, then implement code to make tests pass. This ensures all code is testable and has comprehensive test coverage from the start. > - **Event-Driven Programming**: In Kubernetes controller development, ALWAYS use event-driven programming (e.g., "watch" on kube API) as opposed to polling. Controllers must react to cluster state changes efficiently. > - **Early Returns**: Use as few `else` statements as possible. Return from functions as soon as you can to minimize nesting and improve code clarity (see Early Return / Guard Clause Pattern section). @@ -14,6 +15,25 @@ --- +## 🏛️ GOVERNING METHODOLOGY: Architecture Driven Development (ADD) + +**5-Spot is built ADR-first. Architecture is decided, recorded, and visualized BEFORE code.** For any architecturally significant change, follow this order — full details in `.claude/rules/architecture-driven-development.md`: + +``` +ADR → CALM → TDD → implement → docs +``` + +1. **ADR** — record the decision in `docs/adr/NNNN-title.md` (Status / Context / Decision / Consequences) from `docs/adr/template.md`; update the index in `docs/adr/README.md`. +2. **CALM** — model it in `docs/architecture/calm/architecture.json`; `make calm-validate` + `make calm-diagrams`. *(Process-only decisions have no CALM impact — say so in the ADR and skip this step.)* +3. **TDD** — only then write failing tests, then the minimum implementation (`tdd-workflow` skill); `cargo-quality` gate. +4. **Docs** — CHANGELOG (`**Author:**`) + affected `docs/src/`; `sync-docs`. + +ADRs and CALM diagrams are **first-class deliverables, equal to code and tests.** ADD applies to new CRDs / CRD-field contract changes, controllers/reconcilers/binaries, changes to the CAPI interaction (Machine / bootstrap / infrastructure contract, allowed API groups), deploy/admission/GitOps topology, and cross-cutting security / RBAC / scheduling concerns. Typos, isolated bug fixes, and behavior-preserving refactors are **TDD-only**. **When unsure whether a change is architectural, write the ADR.** + +> See ADR [0001](../docs/adr/0001-adopt-architecture-driven-development.md) for the adoption decision and rationale. + +--- + ## ⚙️ Claude Code Configuration ### 🚨 CRITICAL: Always Verify CRD Schema Sync diff --git a/.claude/SKILL.md b/.claude/SKILL.md index 4a13347..d192920 100644 --- a/.claude/SKILL.md +++ b/.claude/SKILL.md @@ -293,6 +293,12 @@ done **Checklist:** +### If the change is architecturally significant (ADD — do FIRST): +See `.claude/rules/architecture-driven-development.md`. Applies to new CRDs / CRD-field contract changes, controllers/reconcilers/binaries, CAPI-interaction changes, deploy/admission/GitOps topology, and security/RBAC/scheduling concerns. +- [ ] ADR written/updated in `docs/adr/NNNN-*.md` (Status/Context/Decision/Consequences) and indexed in `docs/adr/README.md` +- [ ] CALM model updated, `make calm-validate` + `make calm-diagrams` pass — OR the ADR states "no CALM impact (process-only)" +- [ ] ADR was written BEFORE the implementation (ADR → CALM → TDD order) + ### If ANY `.rs` file was modified: - [ ] Tests updated/added/deleted to match changes (TDD — see `tdd-workflow`) - [ ] All new public functions have tests diff --git a/.claude/rules/architecture-driven-development.md b/.claude/rules/architecture-driven-development.md new file mode 100644 index 0000000..c9853ed --- /dev/null +++ b/.claude/rules/architecture-driven-development.md @@ -0,0 +1,100 @@ +# Architecture Driven Development (ADD) + +> **ADD is the governing methodology for 5-Spot.** Architecture is designed, +> recorded, and visualized **before** code is written. ADRs and CALM diagrams +> are first-class deliverables — equal in importance to the code and the tests. + +ADD layers *on top of* the existing TDD discipline; it does not replace it. The +order is fixed: + +``` +ADR → CALM → TDD → implement → docs +``` + +## The ADD cycle + +For any **architecturally significant** change, complete each step before +starting the next: + +### 1. ADR — decide and record (FIRST) + +Write or update an Architecture Decision Record in +`docs/adr/NNNN-title.md` (lowercase-hyphen, zero-padded sequential number) +using `docs/adr/template.md`, with the standard sections: + +- **Status** — Proposed → Accepted (→ Superseded by NNNN) +- **Context** — the forces, constraints, and the problem being solved +- **Decision** — what we will do, stated plainly +- **Consequences** — trade-offs, follow-ups, what this rules out + +ADRs are kept **in the repo** (unlike roadmaps, which live outside it at +`~/dev/roadmaps/`). One decision per ADR. If a change reverses an earlier ADR, +mark the old one *Superseded* and link forward. Keep the index in +`docs/adr/README.md` current. + +### 2. CALM — model and visualize + +Update the FINOS CALM architecture model +(`docs/architecture/calm/architecture.json`) to reflect the decision: nodes, +relationships, interfaces, controls, and flows. Then: + +```sh +make calm-validate # architecture conforms to the meta-schema (hard gate) +make calm-diagrams # regenerate Mermaid diagrams into docs/src/architecture/ +``` + +The architecture must be modeled and the diagrams must render cleanly **before** +implementation begins. A change that isn't reflected in CALM isn't designed yet. + +> **Process-only decisions** (e.g. adopting a methodology, CI policy) have **no +> CALM impact** — CALM models the running system (controller, `ScheduledMachine` +> CRD, CAPI/provider resources, flows), not the development process. Say so +> explicitly in the ADR's Consequences and skip step 2. + +### 3. TDD — red / green / refactor + +Only now write code, **tests first**, per the `tdd-workflow` skill: failing test +→ minimum implementation → refactor. Tests go in separate `_tests.rs` files +(`src/foo.rs` → `#[cfg(test)] mod foo_tests;` → `src/foo_tests.rs`). After any +`.rs` change, run the `cargo-quality` skill (fmt + clippy + test — all must pass). + +CRD shape changes start in `src/crd.rs` (the source of truth), then regenerate: +`regen-crds` skill (`make crds`) → update `examples/` → `regen-api-docs` +(`make crddoc`, LAST). Never hand-edit `deploy/crds/*.yaml`. + +### 4. Docs + +Update `.claude/CHANGELOG.md` (with `**Author:**`) and any affected `docs/src/` +pages / examples. Run the `sync-docs` skill to verify docs match the code. + +## When does ADD apply? + +**Full ADR + CALM** (architecturally significant): + +- New CRDs / CRD fields that change a contract, new controllers, reconcilers, or binaries +- Changes to how 5-Spot interacts with CAPI (Machine / bootstrap / infrastructure contract, allowed API groups) +- New deploy / admission / GitOps topology (e.g. ValidatingAdmissionPolicy posture, child-cluster routing) +- Cross-cutting concerns: security boundaries, RBAC posture, failure domains, scheduling semantics +- Any decision where "why A over B" is worth recording + +**ADR only, no CALM** (process / policy, no system-topology change): + +- Methodology, CI/CD policy, repository conventions + +**TDD only** (no ADR/CALM needed): + +- Typos, comment/doc tweaks, formatting +- Isolated bug fixes with no architectural impact +- Mechanical refactors that preserve behavior and structure + +> When unsure whether a change is "architectural," **write the ADR.** A short, +> slightly-redundant ADR costs little; an undocumented architectural decision +> costs the next person a re-derivation. + +## Checklist (paste into the work) + +- [ ] ADR written/updated in `docs/adr/NNNN-*.md` (Status/Context/Decision/Consequences); index in `docs/adr/README.md` updated +- [ ] CALM model updated; `make calm-validate` passes; `make calm-diagrams` renders — **or** ADR states "no CALM impact (process-only)" +- [ ] Tests written **first**, then implementation (TDD); CRD changes regenerated via `regen-crds` → `regen-api-docs` +- [ ] `cargo-quality` passes (fmt + clippy + test) +- [ ] CHANGELOG (`**Author:**`) + docs updated; `sync-docs` clean diff --git a/docs/adr/0001-adopt-architecture-driven-development.md b/docs/adr/0001-adopt-architecture-driven-development.md new file mode 100644 index 0000000..1de123d --- /dev/null +++ b/docs/adr/0001-adopt-architecture-driven-development.md @@ -0,0 +1,85 @@ + +# 0001 — Adopt Architecture Driven Development (ADD) + +- **Status:** Accepted +- **Date:** 2026-05-31 +- **Deciders:** Erick Bourgeois +- **Supersedes:** — +- **Related:** `.claude/rules/architecture-driven-development.md`; CALM model at `docs/architecture/calm/architecture.json`; the existing TDD discipline in `.claude/CLAUDE.md`. + +## Context + +5-Spot already enforces strong **Test-Driven Development** (tests first, separate +`_tests.rs` files, `cargo-quality` as a hard gate) and already maintains a +**FINOS CALM** architecture model with `make calm-validate` / `make +calm-diagrams`. What it lacked was an explicit, ordered methodology tying the +*decision*, the *model*, and the *code* together — and a durable home for the +"why A over B" reasoning behind architecturally significant changes. + +Several recent changes illustrate the gap. The RBAC anti-escalation design (VAP +`authorizer` checks for the requesting user + a controller-side +`SelfSubjectAccessReview`), the embedded-`metadata` policy (reject +`name`/`namespace`, allow reserved-prefix-checked `labels`/`annotations`), and +the child-cluster Node-routing model were all real architectural decisions with +non-obvious trade-offs. They were captured in the changelog and code comments, +but there was no first-class record of the decision and the alternatives +weighed — so the next contributor must re-derive the reasoning from the diff. + +A sibling project (banlieue) adopted **Architecture Driven Development**: +decisions are recorded as ADRs and modeled in CALM *before* code, on top of the +existing TDD loop. The order is fixed: + +``` +ADR → CALM → TDD → implement → docs +``` + +The alternative — staying TDD-only and relying on the changelog plus code +comments — keeps the process lighter, but leaves architectural intent scattered +and reconstructed rather than stated. Given 5-Spot operates in a regulated +environment where changes must be auditable and traceable to a rationale, a +first-class decision log is worth the modest per-change overhead. + +## Decision + +We adopt **Architecture Driven Development (ADD)** as the governing methodology +for 5-Spot. For any architecturally significant change, contributors complete +the steps in order — **ADR → CALM → TDD → implement → docs** — before the next +step begins: + +1. **ADR** — record the decision in `docs/adr/NNNN-title.md` (Status / Context / + Decision / Consequences), from `docs/adr/template.md`, indexed in + `docs/adr/README.md`. +2. **CALM** — reflect it in `docs/architecture/calm/architecture.json`; + `make calm-validate` + `make calm-diagrams` must pass. *Process-only decisions + have no CALM impact and say so.* +3. **TDD** — failing tests first, then minimum implementation; `cargo-quality` + gate; CRD changes regenerate via `regen-crds` → `regen-api-docs`. +4. **Docs** — CHANGELOG (`**Author:**`) + affected `docs/src/`; `sync-docs` clean. + +ADRs and CALM diagrams are first-class deliverables, equal to code and tests. +The full rule lives in `.claude/rules/architecture-driven-development.md` and is +referenced as the governing methodology from `.claude/CLAUDE.md`. ADD applies to +new CRDs/CRD-field contract changes, controllers/reconcilers/binaries, changes +to the CAPI interaction, deploy/admission/GitOps topology, and cross-cutting +security/RBAC/scheduling concerns. Typos, isolated bug fixes, and behavior- +preserving refactors remain TDD-only. **When unsure, write the ADR.** + +## Consequences + +- **Easier:** architectural intent is recorded once, at decision time, with the + alternatives weighed — auditable and traceable, which suits the regulated + context. New contributors read `docs/adr/` instead of re-deriving from diffs. +- **Harder / slower:** a modest per-change overhead for significant work (write + the ADR, touch CALM). Mitigated by scoping ADD to *architecturally + significant* changes and keeping ADRs short. +- **Ruled out:** silently making a significant decision in code-only form. If + it's worth a "why A over B," it gets an ADR. +- **Retroactive ADRs:** existing significant decisions (RBAC anti-escalation, + embedded-metadata policy, child-cluster routing, `release:published` docs + trigger) may be back-filled as ADRs over time; not required immediately. +- **CALM impact:** **none.** This is a process decision, not a change to the + running system's topology, so the CALM model is unchanged. (This ADR is itself + an instance of the "process-only → no CALM" rule it establishes.) diff --git a/docs/adr/README.md b/docs/adr/README.md new file mode 100644 index 0000000..423db89 --- /dev/null +++ b/docs/adr/README.md @@ -0,0 +1,39 @@ + +# Architecture Decision Records + +This directory holds 5-Spot's **Architecture Decision Records (ADRs)** — the +durable log of *why* the system is shaped the way it is. + +5-Spot follows **Architecture Driven Development (ADD)**: for any +architecturally significant change, the decision is recorded here **before** code +is written, then modeled in CALM, then implemented test-first. See +[`.claude/rules/architecture-driven-development.md`](../../.claude/rules/architecture-driven-development.md) +for the full methodology. + +## Conventions + +- **Filename:** `NNNN-title.md` — zero-padded sequential number, lowercase, hyphenated. +- **One decision per ADR.** Copy [`template.md`](./template.md) to start. +- **Status lifecycle:** `Proposed` → `Accepted` → (`Superseded by ADR-NNNN`). + Never delete or rewrite a decision — supersede it and link forward. +- **In the repo.** ADRs are version-controlled here. (Roadmaps and phase plans + are *not* — those live outside the repo at `~/dev/roadmaps/`.) + +## When to write one + +Write a full ADR (and update the CALM model) for: new CRDs/CRD-field contract +changes, controllers/reconcilers/binaries, changes to the CAPI interaction +(Machine / bootstrap / infrastructure contract, allowed API groups), deploy / +admission / GitOps topology, and cross-cutting concerns (security boundaries, +RBAC posture, scheduling semantics). Process/policy decisions get an ADR with +**no CALM impact**. Trivial changes (typos, isolated bugfixes, mechanical +refactors) need neither. When unsure, **write the ADR.** + +## Index + +| ADR | Title | Status | +|----:|-------|--------| +| [0001](./0001-adopt-architecture-driven-development.md) | Adopt Architecture Driven Development (ADD) | Accepted | diff --git a/docs/adr/template.md b/docs/adr/template.md new file mode 100644 index 0000000..d55ee0b --- /dev/null +++ b/docs/adr/template.md @@ -0,0 +1,30 @@ + +# NNNN — + +- **Status:** Proposed | Accepted | Superseded by ADR-NNNN +- **Date:** YYYY-MM-DD +- **Deciders:** +- **Supersedes:** +- **Related:** + +## Context + +The forces, constraints, and the problem being solved. What is true today, what +pressure is pushing for a change, and what options were on the table. State the +alternatives you weighed and why they were rejected — that is the part future +readers cannot reconstruct. + +## Decision + +What we will do, stated plainly and in the present tense ("We use X", "Providers +satisfy Y"). One decision per ADR. + +## Consequences + +The trade-offs this locks in: what becomes easier, what becomes harder, what is +now ruled out, and any follow-up work it implies. Note CALM impact explicitly — +either "CALM updated (nodes/flows …)" or "no CALM impact (process-only / no +system-topology change)". diff --git a/examples/workshop/README.md b/examples/workshop/README.md new file mode 100644 index 0000000..deb9af9 --- /dev/null +++ b/examples/workshop/README.md @@ -0,0 +1,288 @@ +# Workshop: Time-Scheduled Worker Nodes with 5-Spot + Cluster API + +A hands-on, fully **local** walkthrough of 5-Spot using upstream **Cluster API** +with the **Docker provider (CAPD)** and the **kubeadm** bootstrap provider. No +cloud account, no SSH hosts, no k0smotron, no k0rdent — everything runs in Docker +on your laptop. + +## The scenario + +You run a dev/CI Kubernetes cluster that only needs extra worker capacity during +working hours. Paying for that worker 24/7 is wasteful. With 5-Spot you declare a +**`ScheduledMachine`** that adds a worker node when a time window opens and drains ++ removes it when the window closes — "spot capacity, on a schedule". + +``` + MANAGEMENT CLUSTER (kind: 5spot-mgmt) + ┌─────────────────────────────────────────────────────────┐ + │ Cluster API core + kubeadm + CAPD │ + │ 5-Spot controller │ + │ │ + │ ScheduledMachine "business-hours-worker" │ + │ │ (window OPEN) │ + │ ▼ │ + │ KubeadmConfig + DockerMachine + Machine ───────────┐│ + └──────────────────────────────────────────────────────────┘│ + ▼ + WORKLOAD CLUSTER "dev-cluster" (CAPD containers) + ┌─────────────────────────────────────────────────────────┐ + │ control-plane node ◄── always on (KubeadmControlPlane) │ + │ business-hours-worker ◄── added/removed by 5-Spot │ + └─────────────────────────────────────────────────────────┘ +``` + +## What you'll learn + +- How 5-Spot turns one `ScheduledMachine` into a CAPI `Machine` + + bootstrap + infrastructure resource. +- How the time schedule drives a real Node joining and leaving a cluster. +- How 5-Spot drains the workload-cluster Node (via the CAPI-generated + `-kubeconfig` Secret) before removing it. +- The kill switch and Node taints. + +## Prerequisites + +| Tool | Notes | +|-------------|-------| +| Docker | Running, with enough headroom for ~4 small containers. | +| `kind` | ≥ v0.24. The CAPD node image tag must match — see below. | +| `kubectl` | Any recent version. | +| `clusterctl`| The Cluster API CLI. **Pin to a release that still serves `cluster.x-k8s.io/v1beta1`** (the v1.8 / v1.9 lines do — 5-Spot emits v1beta1 Machines). | +| `make` | To build + load the 5-Spot image (you run this; the repo never pushes images). | + +> **Version pinning matters.** These manifests use `kindest/node:v1.31.0` +> throughout. Pick a `kindest/node` tag listed for *your* `kind` version, and set +> the **same** tag in `workload-cluster.yaml` (control plane) and +> `scheduledmachine-business-hours.yaml` (worker). + +All commands below are run from this directory (`examples/workshop/`). + +--- + +## Part 1 — Management cluster (kind + Cluster API + 5-Spot) + +**1.1 Create the management cluster** (mounts the Docker socket so CAPD can +create sibling containers): + +```bash +kind create cluster --config kind-management.yaml +# context is now: kind-5spot-mgmt +kubectl cluster-info --context kind-5spot-mgmt +``` + +**1.2 Install Cluster API with the Docker provider:** + +```bash +clusterctl init --infrastructure docker +``` + +Wait until every provider Deployment is `Available`: + +```bash +kubectl wait --for=condition=Available --timeout=300s -n capi-system deploy --all +kubectl wait --for=condition=Available --timeout=300s -n capd-system deploy --all +kubectl wait --for=condition=Available --timeout=300s -n capi-kubeadm-bootstrap-system deploy --all +kubectl wait --for=condition=Available --timeout=300s -n capi-kubeadm-control-plane-system deploy --all +``` + +**1.3 Build, load, and deploy the 5-Spot controller.** From the **repository +root** (you run the image build — the project never builds/pushes images for you): + +```bash +# Build the controller image for your host arch and load it into the kind cluster. +make kind-load KIND_CLUSTER_NAME=5spot-mgmt + +# Install the CRD, the admission policy (RBAC anti-escalation guard), and the controller. +kubectl --context kind-5spot-mgmt apply -f deploy/crds/ +kubectl --context kind-5spot-mgmt apply -R -f deploy/deployment/ +kubectl --context kind-5spot-mgmt apply -f deploy/admission/validatingadmissionpolicy.yaml +kubectl --context kind-5spot-mgmt apply -f deploy/admission/validatingadmissionpolicybinding.yaml + +# Point the Deployment at the image you just loaded and wait for rollout. +# (make kind-load builds and loads ghcr.io/finos/5-spot:local-dev — the KIND_IMAGE.) +kubectl --context kind-5spot-mgmt -n 5spot-system \ + set image deployment/5spot-controller controller=ghcr.io/finos/5-spot:local-dev +kubectl --context kind-5spot-mgmt -n 5spot-system \ + rollout status deployment/5spot-controller --timeout=180s +``` + +> The shipped 5-Spot ClusterRole already grants `create` on +> `bootstrap.cluster.x-k8s.io/*`, `infrastructure.cluster.x-k8s.io/*`, and +> `cluster.x-k8s.io/machines` — so `KubeadmConfig`, `DockerMachine`, and `Machine` +> all pass 5-Spot's pre-flight permission check out of the box. + +Confirm the CRD is healthy: + +```bash +kubectl --context kind-5spot-mgmt get crd scheduledmachines.5spot.finos.org +kubectl --context kind-5spot-mgmt get sm -A # 'sm' is the short name +``` + +--- + +## Part 2 — Workload cluster (CAPD) + +**2.1 Create the workload cluster** (1 control-plane node, no workers yet): + +```bash +kubectl --context kind-5spot-mgmt apply -f workload-cluster.yaml +``` + +Watch it come up (this provisions the LB + control-plane containers — a few minutes): + +```bash +clusterctl --kubeconfig ~/.kube/config describe cluster dev-cluster +kubectl --context kind-5spot-mgmt get kubeadmcontrolplane,machine -w +``` + +Wait for the control plane to initialize: + +```bash +kubectl --context kind-5spot-mgmt wait --for=condition=ControlPlaneInitialized \ + cluster/dev-cluster --timeout=600s +``` + +**2.2 Grab the workload kubeconfig and install a CNI.** CAPD does not ship a CNI, +so nodes stay `NotReady` until you install one: + +```bash +clusterctl get kubeconfig dev-cluster > dev-cluster.kubeconfig + +# Calico (matches the 192.168.0.0/16 pod CIDR in workload-cluster.yaml). +kubectl --kubeconfig dev-cluster.kubeconfig apply -f \ + https://raw.githubusercontent.com/projectcalico/calico/v3.28.0/manifests/calico.yaml + +# The control-plane node should now go Ready. +kubectl --kubeconfig dev-cluster.kubeconfig get nodes -w +``` + +> 5-Spot will **auto-discover** the `dev-cluster-kubeconfig` Secret that CAPI +> created in the `default` namespace (it matches the `-kubeconfig` +> convention) and use it to drain/taint the worker Node later. No extra config. + +--- + +## Part 3 — Schedule a worker with 5-Spot + +The `ScheduledMachine` ships with an **always-on** window so you see the worker +join right away: + +```bash +kubectl --context kind-5spot-mgmt apply -f scheduledmachine-business-hours.yaml +kubectl --context kind-5spot-mgmt get sm business-hours-worker -o wide +``` + +Watch 5-Spot create the three objects on the **management** cluster: + +```bash +kubectl --context kind-5spot-mgmt get kubeadmconfig,dockermachine,machine \ + -l 5spot.finos.org/scheduled-machine=business-hours-worker +``` + +Then watch the worker Node appear on the **workload** cluster and go Ready: + +```bash +kubectl --kubeconfig dev-cluster.kubeconfig get nodes -w +# business-hours-worker Ready ... +``` + +Confirm the schedule status and the taint 5-Spot applied: + +```bash +kubectl --context kind-5spot-mgmt get sm business-hours-worker \ + -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,READY:.status.ready,IN_SCHEDULE:.status.inSchedule +kubectl --kubeconfig dev-cluster.kubeconfig get node business-hours-worker \ + -o jsonpath='{.spec.taints}{"\n"}' +``` + +--- + +## Part 4 — Watch scheduling actually schedule + +**4a. Close the window** by disabling the schedule. 5-Spot cordons + drains the +Node, then deletes the Machine/DockerMachine/KubeadmConfig: + +```bash +kubectl --context kind-5spot-mgmt patch sm business-hours-worker \ + --type merge -p '{"spec":{"schedule":{"enabled":false}}}' + +# Phase moves Active -> ShuttingDown -> Inactive; the Node drains and disappears. +kubectl --context kind-5spot-mgmt get sm business-hours-worker -w +kubectl --kubeconfig dev-cluster.kubeconfig get nodes -w +``` + +**4b. Re-open the window** — the worker comes back: + +```bash +kubectl --context kind-5spot-mgmt patch sm business-hours-worker \ + --type merge -p '{"spec":{"schedule":{"enabled":true}}}' +kubectl --kubeconfig dev-cluster.kubeconfig get nodes -w +``` + +**4c. Try a real time window.** Edit the SM and set a window that does **not** +include "now" (in your timezone) to prove time-based removal, e.g. a past hour +range, then set it back. This is what `daysOfWeek: ["mon-fri"]` / +`hoursOfDay: ["9-17"]` does in production — outside the window, the worker is +gone (and you stop paying for it). + +--- + +## Part 5 (bonus) — The kill switch + +`killSwitch: true` removes the worker **immediately**, bypassing the graceful +window — your "get it out now" lever: + +```bash +kubectl --context kind-5spot-mgmt patch sm business-hours-worker \ + --type merge -p '{"spec":{"killSwitch":true}}' +# ...then flip it back to false to let the schedule resume. +``` + +--- + +## Troubleshooting + +- **Worker stuck `Provisioning`** — check the bootstrap and infra objects: + ```bash + kubectl --context kind-5spot-mgmt describe machine business-hours-worker + kubectl --context kind-5spot-mgmt describe dockermachine business-hours-worker + kubectl --context kind-5spot-mgmt logs -n capd-system deploy/capd-controller-manager + ``` +- **`ScheduledMachine` rejected at apply time** — the admission policy enforces + that *you* (the applying user) can `create` the embedded `KubeadmConfig` / + `DockerMachine`, and forbids `metadata.name` / `metadata.namespace` on them. + Running as cluster-admin (the kind default) satisfies the permission check. +- **Node never goes Ready** — you skipped the CNI (Part 2.2), or the + `kindest/node` tag doesn't match your `kind` version. +- **5-Spot can't drain the Node** — confirm the Secret exists and is discovered: + ```bash + kubectl --context kind-5spot-mgmt get secret dev-cluster-kubeconfig + kubectl --context kind-5spot-mgmt logs -n 5spot-system deploy/5spot-controller | grep -i kubeconfig + ``` +- **Image pull errors for the worker** — pre-pull the node image and load it: + `docker pull kindest/node:v1.31.0 && kind load docker-image kindest/node:v1.31.0 --name 5spot-mgmt`. + +## Teardown + +```bash +./teardown.sh +``` + +--- + +## How this maps to production + +The only things that change for a real environment are the **providers** behind +`bootstrapSpec` and `infrastructureSpec`: + +| Workshop (here) | Production examples | +|----------------------------|-----------------------------------------------------| +| `DockerMachine` (CAPD) | `AWSMachine`, `AzureMachine`, `Metal3Machine`, `RemoteMachine`, … | +| `KubeadmConfig` | `KubeadmConfig`, or another allowed bootstrap provider | +| `kindest/node` customImage | a real AMI / image / bare-metal host | + +The `ScheduledMachine` shape — `schedule`, `clusterName`, `machineTemplate`, +`nodeTaints`, `gracefulShutdownTimeout`, `killSwitch` — is identical. Swap the +embedded specs for your provider's types (whose API groups must be in 5-Spot's +allowlist: `bootstrap.cluster.x-k8s.io`, `infrastructure.cluster.x-k8s.io`) and +the same schedule logic provisions and reclaims real machines. diff --git a/examples/workshop/kind-management.yaml b/examples/workshop/kind-management.yaml new file mode 100644 index 0000000..b4f0db3 --- /dev/null +++ b/examples/workshop/kind-management.yaml @@ -0,0 +1,20 @@ +# Kind configuration for the 5-Spot WORKSHOP management cluster. +# +# This single kind cluster hosts: +# - Cluster API core + the kubeadm bootstrap/control-plane providers +# - The Cluster API Docker infrastructure provider (CAPD) +# - The 5-Spot controller +# +# CAPD provisions "machines" as sibling Docker containers, so the kind node +# needs access to the host Docker socket. That single extraMount is the only +# special requirement. +kind: Cluster +apiVersion: kind.x-k8s.io/v1alpha4 +name: 5spot-mgmt +nodes: + - role: control-plane + extraMounts: + # CAPD talks to the host Docker daemon to create workload-cluster + # containers as siblings of this node. + - hostPath: /var/run/docker.sock + containerPath: /var/run/docker.sock diff --git a/examples/workshop/scheduledmachine-business-hours.yaml b/examples/workshop/scheduledmachine-business-hours.yaml new file mode 100644 index 0000000..0342709 --- /dev/null +++ b/examples/workshop/scheduledmachine-business-hours.yaml @@ -0,0 +1,74 @@ +# 5-Spot ScheduledMachine — a time-scheduled WORKER for the CAPD dev-cluster. +# +# When the schedule window is OPEN, 5-Spot creates three objects on the +# management cluster, all named after this ScheduledMachine (business-hours-worker): +# 1. KubeadmConfig (bootstrap.cluster.x-k8s.io) — generates the kubeadm join +# 2. DockerMachine (infrastructure.cluster.x-k8s.io) — the worker container +# 3. Machine (cluster.x-k8s.io) — ties the two to the Cluster +# When the window CLOSES (or enabled:false / killSwitch), 5-Spot drains the Node +# (via the auto-discovered dev-cluster-kubeconfig Secret) and deletes all three. +apiVersion: 5spot.finos.org/v1alpha1 +kind: ScheduledMachine +metadata: + name: business-hours-worker + namespace: default +spec: + # MUST match the CAPI Cluster name in workload-cluster.yaml. 5-Spot also uses + # this to auto-discover the `-kubeconfig` Secret that CAPI creates, + # which it uses to drain/taint the Node on the workload cluster. + clusterName: dev-cluster + + schedule: + # ── WORKSHOP setting: always-on, so the worker joins immediately. ───────── + daysOfWeek: ["mon-sun"] + hoursOfDay: ["0-23"] + # ── Real "business hours" setting — uncomment to only run Mon–Fri 9am–5pm: ─ + # daysOfWeek: ["mon-fri"] + # hoursOfDay: ["9-17"] + timezone: "UTC" + enabled: true + + # Bootstrap: a kubeadm worker join config. 5-Spot creates this as a concrete + # KubeadmConfig (not a template) named business-hours-worker. + bootstrapSpec: + apiVersion: bootstrap.cluster.x-k8s.io/v1beta1 + kind: KubeadmConfig + spec: + joinConfiguration: + nodeRegistration: + criSocket: unix:///var/run/containerd/containerd.sock + kubeletExtraArgs: + eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0% + + # Infrastructure: a CAPD worker container. + # + # customImage is REQUIRED here: 5-Spot does not set `Machine.spec.version`, so + # CAPD cannot derive the node image from the Kubernetes version — you must pin + # it. Use the SAME tag as the control plane in workload-cluster.yaml. + infrastructureSpec: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DockerMachine + spec: + customImage: kindest/node:v1.31.0 + + # Optional labels/annotations stamped on the CAPI Machine (and, separately, + # any embedded-resource labels via bootstrapSpec.metadata / infrastructureSpec.metadata). + # Reserved prefixes (kubernetes.io/, k8s.io/, cluster.x-k8s.io/, 5spot.finos.org/) + # are rejected, so use your own domain. + machineTemplate: + labels: + workshop.example.com/pool: spot + annotations: + workshop.example.com/owner: platform-team + + # Apply a taint to the scheduled Node once it is Ready, so only workloads that + # tolerate "spot" capacity land on it. 5-Spot reconciles only the taints it sets. + nodeTaints: + - key: workshop.example.com/spot + value: "true" + effect: NoSchedule + + priority: 50 + gracefulShutdownTimeout: 5m + nodeDrainTimeout: 5m + killSwitch: false diff --git a/examples/workshop/teardown.sh b/examples/workshop/teardown.sh new file mode 100755 index 0000000..9f2fc36 --- /dev/null +++ b/examples/workshop/teardown.sh @@ -0,0 +1,25 @@ +#!/usr/bin/env bash +# Tear down the 5-Spot workshop environment. +# +# Order matters: delete the ScheduledMachine first so 5-Spot drains the Node and +# removes its CAPI objects cleanly, then delete the workload Cluster (CAPD +# removes its containers), then delete the management kind cluster. +set -euo pipefail + +MGMT_CONTEXT="kind-5spot-mgmt" +KIND_CLUSTER="5spot-mgmt" + +echo "==> Removing the ScheduledMachine (lets 5-Spot drain + clean up the worker)" +kubectl --context "${MGMT_CONTEXT}" delete -f scheduledmachine-business-hours.yaml --ignore-not-found +# Give the finalizer a moment to drain the node and delete the Machine/Docker/Kubeadm objects. +sleep 15 + +echo "==> Deleting the workload Cluster (CAPD tears down its containers)" +kubectl --context "${MGMT_CONTEXT}" delete -f workload-cluster.yaml --ignore-not-found --wait=false || true +# Wait for the Cluster object to be fully removed so CAPD finishes container cleanup. +kubectl --context "${MGMT_CONTEXT}" wait --for=delete cluster/dev-cluster -n default --timeout=180s || true + +echo "==> Deleting the management kind cluster" +kind delete cluster --name "${KIND_CLUSTER}" + +echo "==> Done. Run 'docker ps' to confirm no stray 'dev-cluster-*' containers remain." diff --git a/examples/workshop/workload-cluster.yaml b/examples/workshop/workload-cluster.yaml new file mode 100644 index 0000000..0f57775 --- /dev/null +++ b/examples/workshop/workload-cluster.yaml @@ -0,0 +1,92 @@ +# Workshop WORKLOAD cluster — plain Cluster API with the Docker provider (CAPD). +# +# This defines a 1-node control plane only. It deliberately has NO worker +# MachineDeployment: the worker capacity is supplied on a time schedule by the +# 5-Spot `ScheduledMachine` (see scheduledmachine-business-hours.yaml). +# +# Everything here is upstream Cluster API — no k0smotron, no k0rdent. +# +# NOTE ON API VERSION: the 5-Spot controller currently creates CAPI `Machine` +# objects under `cluster.x-k8s.io/v1beta1`. Install a Cluster API release that +# still serves the v1beta1 API (the v1.8 / v1.9 lines do). These manifests use +# v1beta1 throughout for consistency. +# +# NOTE ON THE NODE IMAGE: pin `kindest/node` to a tag that matches your local +# `kind` version (see the kind release notes). The same tag is reused by the +# worker in scheduledmachine-business-hours.yaml. +--- +apiVersion: cluster.x-k8s.io/v1beta1 +kind: Cluster +metadata: + name: dev-cluster + namespace: default +spec: + clusterNetwork: + pods: + cidrBlocks: ["192.168.0.0/16"] # matches the Calico install in the README + services: + cidrBlocks: ["10.128.0.0/12"] + serviceDomain: cluster.local + controlPlaneRef: + apiVersion: controlplane.cluster.x-k8s.io/v1beta1 + kind: KubeadmControlPlane + name: dev-cluster-control-plane + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DockerCluster + name: dev-cluster +--- +# CAPD manages a HAProxy load-balancer container in front of the control plane. +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DockerCluster +metadata: + name: dev-cluster + namespace: default +spec: {} +--- +# Infrastructure template for the control-plane Machine(s). +apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 +kind: DockerMachineTemplate +metadata: + name: dev-cluster-control-plane + namespace: default +spec: + template: + spec: + # Pin the image so the workshop is reproducible. Match your kind version. + customImage: kindest/node:v1.31.0 +--- +apiVersion: controlplane.cluster.x-k8s.io/v1beta1 +kind: KubeadmControlPlane +metadata: + name: dev-cluster-control-plane + namespace: default +spec: + replicas: 1 + version: v1.31.0 + machineTemplate: + infrastructureRef: + apiVersion: infrastructure.cluster.x-k8s.io/v1beta1 + kind: DockerMachineTemplate + name: dev-cluster-control-plane + kubeadmConfigSpec: + # Standard CAPD control-plane settings (mirrors the clusterctl "docker" + # development flavor). + clusterConfiguration: + apiServer: + certSANs: [localhost, 127.0.0.1, 0.0.0.0, host.docker.internal] + controllerManager: + extraArgs: + enable-hostpath-provisioner: "true" + initConfiguration: + nodeRegistration: + criSocket: unix:///var/run/containerd/containerd.sock + kubeletExtraArgs: + # kind nodes run on a shared host; relax disk-pressure eviction so the + # kubelet does not evict everything under normal laptop disk usage. + eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0% + joinConfiguration: + nodeRegistration: + criSocket: unix:///var/run/containerd/containerd.sock + kubeletExtraArgs: + eviction-hard: nodefs.available<0%,nodefs.inodesFree<0%,imagefs.available<0%