Skip to content

Commit d3deb04

Browse files
ebourgeoisebourgeois
committed
Add leader election for HA
Behaviour when ENABLE_LEADER_ELECTION=true: 1. All replicas start with is_leader = false 2. Background task races to acquire the 5spot-leader Lease 3. Winner sets is_leader = true and starts reconciling 4. Losers return await_change() for every event — no API pressure, instant response when they do take over 5. If the leader fails, a non-leader acquires the lease within LEASE_DURATION_SECONDS (15 s default) and takes over Change SPDX liceense headers to Apache Signed-off-by: Erick Bourgeois <erick@jeb.ca>
1 parent 5e5a9ec commit d3deb04

47 files changed

Lines changed: 2641 additions & 1463 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.claude/CHANGELOG.md‎

Lines changed: 155 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,160 @@ The format is based on the regulated environment requirements:
99

1010
---
1111

12+
## [2026-04-10 08:30] - Consolidate pr.yaml, main.yaml, release.yaml into single build.yaml
13+
14+
**Author:** Erick Bourgeois
15+
16+
### Changed
17+
- `.github/workflows/build.yaml`: New consolidated workflow replacing the three separate files; triggers on `pull_request`, `push` to main, and `release: published`; uses `if:` at job and step level to gate event-specific behaviour
18+
- `.github/workflows/pr.yaml`: Deleted
19+
- `.github/workflows/main.yaml`: Deleted
20+
- `.github/workflows/release.yaml`: Deleted
21+
22+
### Why
23+
Three workflows shared the same build matrix, env vars, and most job logic, requiring the same fix to be applied in three places (e.g. the linker override, the attest job). A single file is easier to maintain and gives a complete picture of CI behaviour in one place.
24+
25+
### Impact
26+
- [ ] Breaking change
27+
- [ ] Requires cluster rollout
28+
- [ ] Config change only
29+
- [ ] Documentation only
30+
31+
> **Key design decisions:**
32+
> - `extract-version` serves as the quality gate: it runs after all checks that apply to the current event (`verify-commits`, `license-check`, `format`) and all downstream jobs depend on it.
33+
> - Docker metadata uses three separate `docker/metadata-action` steps gated by `if:`; `docker/build-push-action` concatenates all outputs and filters empty lines.
34+
> - Cosign signing, Docker SBOM generation, `sign-artifacts`, SLSA provenance, and `upload-release-assets` are guarded by `if: github.event_name == 'release'`.
35+
> - `test` and `format`/`clippy` are guarded by `if: github.event_name == 'pull_request'`.
36+
> - `trivy` is guarded by `if: github.event_name != 'pull_request'`.
37+
> - Artifact retention: PR/push = 1 day (two upload steps); release = default.
38+
39+
---
40+
41+
## [2026-04-10 07:45] - Add GitHub artifact attestation job to all three CI/CD workflows
42+
43+
**Author:** Erick Bourgeois
44+
45+
### Changed
46+
- `.github/workflows/pr.yaml`: Added `id: docker_build` to docker build step; added `outputs:` block to docker job (per-variant digests); added `export-digest` step; added `attest` job depending on `docker` and `extract-version`
47+
- `.github/workflows/main.yaml`: Same additions to docker job and new `attest` job
48+
- `.github/workflows/release.yaml`: Same additions to `docker-release` job and new `attest` job (depends on `docker-release`)
49+
50+
### Why
51+
GitHub's `actions/attest-build-provenance` generates a signed SLSA provenance attestation stored natively in GitHub Artifact Attestations and optionally pushed to the OCI registry alongside the image. This is queryable with `gh attestation verify` and complements the existing Cosign signatures in the release workflow. Requires the matrix digest-export pattern to pass per-image digests from a matrix job to a downstream job.
52+
53+
### Impact
54+
- [ ] Breaking change
55+
- [ ] Requires cluster rollout
56+
- [ ] Config change only
57+
- [x] Documentation only
58+
59+
---
60+
61+
## [2026-04-08 17:30] - Add documentation build and GitHub Pages deployment workflow
62+
63+
**Author:** Erick Bourgeois
64+
65+
### Changed
66+
- `.github/workflows/docs.yaml`: New workflow — builds MkDocs documentation (including `make docs` which runs `cargo run --bin crddoc`) and deploys to GitHub Pages on push to main; runs link checks on PRs
67+
68+
### Why
69+
The project has a full MkDocs documentation site under `docs/` but no automated build or publishing pipeline. This workflow closes that gap by building on every relevant change, checking for broken links on PRs, and publishing to GitHub Pages on every merge to main.
70+
71+
### Impact
72+
- [ ] Breaking change
73+
- [ ] Requires cluster rollout
74+
- [ ] Config change only
75+
- [x] Documentation only
76+
77+
> **Note:** GitHub Pages must be enabled in the repository settings (`Settings → Pages → Source: GitHub Actions`) for the deploy job to succeed. The `poetry.lock` file should be committed after the first `poetry install` run to improve cache efficiency and build reproducibility.
78+
79+
---
80+
81+
## [2026-04-09 11:00] - Fix CI linker error caused by .cargo/config.toml on Linux runners
82+
83+
**Author:** Erick Bourgeois
84+
85+
### Changed
86+
- `.github/workflows/pr.yaml`: Added `CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER: cc` and `CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER: cc` to the top-level `env:` block
87+
- `.github/workflows/main.yaml`: Same
88+
- `.github/workflows/release.yaml`: Same
89+
90+
### Why
91+
`.cargo/config.toml` specifies `linker = "x86_64-unknown-linux-gnu-gcc"` and `linker = "aarch64-unknown-linux-gnu-gcc"` for the respective targets — Homebrew cross-compilers needed when a macOS developer uses `cargo build --target <linux-triple>` locally (the Makefile fallback path). On Linux CI runners, `cargo build --release` with no explicit target resolves to the **native** triple (e.g., `x86_64-unknown-linux-gnu` on `ubuntu-latest`), which picks up the same override and fails because the cross-compiler is not installed on GitHub Actions runners. `CARGO_TARGET_*_LINKER` environment variables take precedence over `config.toml`, restoring `cc` (the system linker) in CI without modifying the config file.
92+
93+
### Impact
94+
- [ ] Breaking change
95+
- [ ] Requires cluster rollout
96+
- [ ] Config change only
97+
- [ ] Documentation only
98+
99+
---
100+
101+
## [2026-04-09 10:00] - Replace cross-compilation with native cargo builds in CI
102+
103+
**Author:** Erick Bourgeois
104+
105+
### Changed
106+
- `.github/workflows/pr.yaml`: Replaced `firestoned/github-actions/rust/setup-rust-build@v1.3.6` + `build-binary@v1.3.6` + `generate-sbom@v1.3.6` with `dtolnay/rust-toolchain@stable` + `cargo build --release` + `cargo-cyclonedx`; switched ARM64 build to `ubuntu-24.04-arm` native runner; updated artifact paths from `target/$target/release/` to `target/release/`; fixed `license-id: "MIT"` → `"Apache-2.0"`
107+
- `.github/workflows/main.yaml`: Same build and license-check changes
108+
- `.github/workflows/release.yaml`: Same build and license-check changes; replaced `setup-rust-build` in `package-deploy-manifests` job with `dtolnay/rust-toolchain@stable`
109+
110+
### Why
111+
The `firestoned/github-actions/rust/build-binary@v1.3.6` action internally sets `-C linker=x86_64-unknown-linux-gnu-gcc`, which is not installed on GitHub Actions `ubuntu-latest` runners, causing all build jobs to fail. Native `cargo build --release` on arch-appropriate runners eliminates cross-compilation entirely. License-id was stale after the FINOS Apache-2.0 migration.
112+
113+
### Impact
114+
- [ ] Breaking change
115+
- [ ] Requires cluster rollout
116+
- [ ] Config change only
117+
- [x] Documentation only
118+
119+
---
120+
121+
## [2026-04-09 03:00] - Phase 2 (P2-4): leader election via kube-lease-manager
122+
123+
**Author:** Erick Bourgeois
124+
125+
### Changed
126+
- `Cargo.toml`: Added `kube-lease-manager = "0.11"` dependency
127+
- `src/constants.rs`: Added `DEFAULT_LEASE_NAME`, `DEFAULT_LEASE_DURATION_SECS`, `DEFAULT_LEASE_RENEW_DEADLINE_SECS`, `DEFAULT_LEASE_RETRY_PERIOD_SECS`, `DEFAULT_LEASE_GRACE_SECS`, `DEFAULT_LEASE_NAMESPACE` constants
128+
- `src/reconcilers/scheduled_machine.rs`: Added `is_leader: Arc<AtomicBool>` to `Context` (defaults to `true` for backward-compatible single-instance mode); added leader guard in `reconcile_guarded` — non-leaders return `Action::await_change()` immediately
129+
- `src/main.rs`: Added `enable_leader_election`, `lease_name`, `lease_namespace`, `lease_duration_secs`, `lease_renew_deadline_secs` CLI args; when enabled, sets `is_leader = false` at startup and spawns a background `kube-lease-manager` task that flips `is_leader` on acquisition/loss
130+
- `deploy/deployment/deployment.yaml`: Fixed `POD_NAME` → `CONTROLLER_POD_NAME` env var (aligns with `Context::new` and leader election holder identity)
131+
- `src/reconcilers/scheduled_machine_tests.rs`: Added 2 TDD tests — `test_context_new_defaults_is_leader_to_true` and `test_reconcile_guarded_awaits_change_when_not_leader`
132+
- `docs/src/operations/configuration.md`: Added all leader election env vars, CLI args, Leader Election section, Lease RBAC rules
133+
134+
### Why
135+
Basel III HA (P2-4): a single-replica controller is a single point of failure. With `ENABLE_LEADER_ELECTION=true` and `replicas: 2`, only the lease holder reconciles resources. Standby replicas react within one `LEASE_DURATION_SECONDS` window on leader failure. `Context::is_leader` defaults to `true` so existing single-replica deployments continue without any config change.
136+
137+
### Impact
138+
- [ ] Breaking change
139+
- [x] Requires cluster rollout — set `ENABLE_LEADER_ELECTION=true` and `replicas: 2`; RBAC for `leases` already in `clusterrole.yaml`
140+
- [ ] Config change only
141+
- [ ] Documentation only
142+
143+
---
144+
145+
## [2026-04-09 02:00] - Add SPDX license headers to all .github YAML files
146+
147+
**Author:** Erick Bourgeois
148+
149+
### Changed
150+
- `.github/ISSUE_TEMPLATE/bug_report.yml`: Added `# Copyright (c) 2025 Erick Bourgeois, finos` + `# SPDX-License-Identifier: Apache-2.0` header
151+
- `.github/ISSUE_TEMPLATE/feature_request.yml`: Same
152+
- `.github/ISSUE_TEMPLATE/meeting_minutes.yml`: Same
153+
- `.github/ISSUE_TEMPLATE/support_question.yml`: Same
154+
155+
### Why
156+
Supply-chain provenance and automated license scanning (NIST SA-4) require SPDX headers on all project-owned files. The three workflow files and both composite actions already had headers from P2-10; these four issue templates were the remaining `.github/` YAML files without them. `dco.yml` was intentionally left untouched — it is managed by FINOS and carries an explicit "Do not edit" notice.
157+
158+
### Impact
159+
- [ ] Breaking change
160+
- [ ] Requires cluster rollout
161+
- [ ] Config change only
162+
- [x] Documentation only
163+
164+
---
165+
12166
## [2026-04-09 01:00] - Phase 2 (P2-5): exponential back-off in error policy
13167

14168
**Author:** Erick Bourgeois
@@ -86,7 +240,7 @@ The `ValidatingAdmissionPolicy` deployed in the previous entry had no user-facin
86240
- `src/**/*.rs` (all 18 files): Added P2-10 SPDX supply-chain provenance headers to every Rust source file:
87241
```
88242
// Copyright (c) 2025 Erick Bourgeois, RBC Capital Markets
89-
// SPDX-License-Identifier: MIT
243+
// SPDX-License-Identifier: Apache-2.0
90244
```
91245

92246
### Why

‎.github/ISSUE_TEMPLATE/bug_report.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
name: 🐛 Bug Report
24
description: If something isn't working as expected 🤔
35
title: "[Bug] "

‎.github/ISSUE_TEMPLATE/feature_request.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
name: 🚀 Feature Request
24
description: I have a suggestion (and may want to implement it 🙂)!
35
title: "[Feature] "

‎.github/ISSUE_TEMPLATE/meeting_minutes.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
name: 🐛 Bug Report
24
description: If something isn't working as expected 🤔
35
title: "[Bug] "

‎.github/ISSUE_TEMPLATE/support_question.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
name: 🤗 Support Question
24
description: If you have a question about configuration, usage, etc. 💬
35
title: "[Support] "

‎.github/actions/extract-version/action.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Copyright (c) 2025 Erick Bourgeois, finos
2-
# SPDX-License-Identifier: MIT
2+
# SPDX-License-Identifier: Apache-2.0
33

44
name: 'Extract Version Information'
55
description: 'Extracts version, tag name, and image tags for consistent use across workflows'

‎.github/actions/prepare-docker-binaries/action.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Copyright (c) 2025 Erick Bourgeois, finos
2-
# SPDX-License-Identifier: MIT
2+
# SPDX-License-Identifier: Apache-2.0
33

44
name: 'Prepare Docker Binaries'
55
description: 'Downloads artifacts and prepares binaries for multi-arch Docker build'

‎.github/dco.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
# Do not edit this file! Email help@finos.org if you have any questions.
24

35
allowRemediationCommits:

0 commit comments

Comments
 (0)