Skip to content

Commit f07d132

Browse files
authored
Add leader election for HA (#7)
Behaviour when ENABLE_LEADER_ELECTION=true: 1. All replicas start with is_leader = false 2. Background task races to acquire the 5spot-leader Lease 3. Winner sets is_leader = true and starts reconciling 4. Losers return await_change() for every event — no API pressure, instant response when they do take over 5. If the leader fails, a non-leader acquires the lease within LEASE_DURATION_SECONDS (15 s default) and takes over Change SPDX liceense headers to Apache Signed-off-by: Erick Bourgeois <erick@jeb.ca>
1 parent 5e5a9ec commit f07d132

48 files changed

Lines changed: 2733 additions & 1490 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.claude/CHANGELOG.md

Lines changed: 191 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,196 @@ The format is based on the regulated environment requirements:
99

1010
---
1111

12+
## [2026-04-10 08:50] - Extend Cosign image signing to main-branch pushes
13+
14+
**Author:** Erick Bourgeois
15+
16+
### Changed
17+
- `.github/workflows/build.yaml`: Changed Cosign signing step condition from `github.event_name == 'release'` to `github.event_name != 'pull_request'`
18+
19+
### Why
20+
Main-branch images are tagged `latest` and `main-YYYY-MM-DD` and may be deployed to staging. Signing them allows `cosign verify` to work on staging images, not just production releases. PR images remain unsigned — they are ephemeral, tagged `pr-{number}`, and not deployed anywhere.
21+
22+
### Impact
23+
- [ ] Breaking change
24+
- [ ] Requires cluster rollout
25+
- [ ] Config change only
26+
- [ ] Documentation only
27+
28+
---
29+
30+
## [2026-04-10 08:45] - Fix attest job: add GHCR login before push-to-registry
31+
32+
**Author:** Erick Bourgeois
33+
34+
### Changed
35+
- `.github/workflows/build.yaml`: Added `docker/login-action@v3` step to the `attest` job before `actions/attest-build-provenance@v2`
36+
37+
### Why
38+
`push-to-registry: true` in `actions/attest-build-provenance` pushes the attestation bundle as an OCI artifact to GHCR, which requires registry credentials. Each job runs in a fresh environment — the Docker login performed by `firestoned/github-actions/docker/setup-docker` in the `docker` job does not carry over to the `attest` job.
39+
40+
### Impact
41+
- [ ] Breaking change
42+
- [ ] Requires cluster rollout
43+
- [ ] Config change only
44+
- [ ] Documentation only
45+
46+
---
47+
48+
## [2026-04-10 08:30] - Consolidate pr.yaml, main.yaml, release.yaml into single build.yaml
49+
50+
**Author:** Erick Bourgeois
51+
52+
### Changed
53+
- `.github/workflows/build.yaml`: New consolidated workflow replacing the three separate files; triggers on `pull_request`, `push` to main, and `release: published`; uses `if:` at job and step level to gate event-specific behaviour
54+
- `.github/workflows/pr.yaml`: Deleted
55+
- `.github/workflows/main.yaml`: Deleted
56+
- `.github/workflows/release.yaml`: Deleted
57+
58+
### Why
59+
Three workflows shared the same build matrix, env vars, and most job logic, requiring the same fix to be applied in three places (e.g. the linker override, the attest job). A single file is easier to maintain and gives a complete picture of CI behaviour in one place.
60+
61+
### Impact
62+
- [ ] Breaking change
63+
- [ ] Requires cluster rollout
64+
- [ ] Config change only
65+
- [ ] Documentation only
66+
67+
> **Key design decisions:**
68+
> - `extract-version` serves as the quality gate: it runs after all checks that apply to the current event (`verify-commits`, `license-check`, `format`) and all downstream jobs depend on it.
69+
> - Docker metadata uses three separate `docker/metadata-action` steps gated by `if:`; `docker/build-push-action` concatenates all outputs and filters empty lines.
70+
> - Cosign signing, Docker SBOM generation, `sign-artifacts`, SLSA provenance, and `upload-release-assets` are guarded by `if: github.event_name == 'release'`.
71+
> - `test` and `format`/`clippy` are guarded by `if: github.event_name == 'pull_request'`.
72+
> - `trivy` is guarded by `if: github.event_name != 'pull_request'`.
73+
> - Artifact retention: PR/push = 1 day (two upload steps); release = default.
74+
75+
---
76+
77+
## [2026-04-10 07:45] - Add GitHub artifact attestation job to all three CI/CD workflows
78+
79+
**Author:** Erick Bourgeois
80+
81+
### Changed
82+
- `.github/workflows/pr.yaml`: Added `id: docker_build` to docker build step; added `outputs:` block to docker job (per-variant digests); added `export-digest` step; added `attest` job depending on `docker` and `extract-version`
83+
- `.github/workflows/main.yaml`: Same additions to docker job and new `attest` job
84+
- `.github/workflows/release.yaml`: Same additions to `docker-release` job and new `attest` job (depends on `docker-release`)
85+
86+
### Why
87+
GitHub's `actions/attest-build-provenance` generates a signed SLSA provenance attestation stored natively in GitHub Artifact Attestations and optionally pushed to the OCI registry alongside the image. This is queryable with `gh attestation verify` and complements the existing Cosign signatures in the release workflow. Requires the matrix digest-export pattern to pass per-image digests from a matrix job to a downstream job.
88+
89+
### Impact
90+
- [ ] Breaking change
91+
- [ ] Requires cluster rollout
92+
- [ ] Config change only
93+
- [x] Documentation only
94+
95+
---
96+
97+
## [2026-04-08 17:30] - Add documentation build and GitHub Pages deployment workflow
98+
99+
**Author:** Erick Bourgeois
100+
101+
### Changed
102+
- `.github/workflows/docs.yaml`: New workflow — builds MkDocs documentation (including `make docs` which runs `cargo run --bin crddoc`) and deploys to GitHub Pages on push to main; runs link checks on PRs
103+
104+
### Why
105+
The project has a full MkDocs documentation site under `docs/` but no automated build or publishing pipeline. This workflow closes that gap by building on every relevant change, checking for broken links on PRs, and publishing to GitHub Pages on every merge to main.
106+
107+
### Impact
108+
- [ ] Breaking change
109+
- [ ] Requires cluster rollout
110+
- [ ] Config change only
111+
- [x] Documentation only
112+
113+
> **Note:** GitHub Pages must be enabled in the repository settings (`Settings → Pages → Source: GitHub Actions`) for the deploy job to succeed. The `poetry.lock` file should be committed after the first `poetry install` run to improve cache efficiency and build reproducibility.
114+
115+
---
116+
117+
## [2026-04-09 11:00] - Fix CI linker error caused by .cargo/config.toml on Linux runners
118+
119+
**Author:** Erick Bourgeois
120+
121+
### Changed
122+
- `.github/workflows/pr.yaml`: Added `CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER: cc` and `CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER: cc` to the top-level `env:` block
123+
- `.github/workflows/main.yaml`: Same
124+
- `.github/workflows/release.yaml`: Same
125+
126+
### Why
127+
`.cargo/config.toml` specifies `linker = "x86_64-unknown-linux-gnu-gcc"` and `linker = "aarch64-unknown-linux-gnu-gcc"` for the respective targets — Homebrew cross-compilers needed when a macOS developer uses `cargo build --target <linux-triple>` locally (the Makefile fallback path). On Linux CI runners, `cargo build --release` with no explicit target resolves to the **native** triple (e.g., `x86_64-unknown-linux-gnu` on `ubuntu-latest`), which picks up the same override and fails because the cross-compiler is not installed on GitHub Actions runners. `CARGO_TARGET_*_LINKER` environment variables take precedence over `config.toml`, restoring `cc` (the system linker) in CI without modifying the config file.
128+
129+
### Impact
130+
- [ ] Breaking change
131+
- [ ] Requires cluster rollout
132+
- [ ] Config change only
133+
- [ ] Documentation only
134+
135+
---
136+
137+
## [2026-04-09 10:00] - Replace cross-compilation with native cargo builds in CI
138+
139+
**Author:** Erick Bourgeois
140+
141+
### Changed
142+
- `.github/workflows/pr.yaml`: Replaced `firestoned/github-actions/rust/setup-rust-build@v1.3.6` + `build-binary@v1.3.6` + `generate-sbom@v1.3.6` with `dtolnay/rust-toolchain@stable` + `cargo build --release` + `cargo-cyclonedx`; switched ARM64 build to `ubuntu-24.04-arm` native runner; updated artifact paths from `target/$target/release/` to `target/release/`; fixed `license-id: "MIT"``"Apache-2.0"`
143+
- `.github/workflows/main.yaml`: Same build and license-check changes
144+
- `.github/workflows/release.yaml`: Same build and license-check changes; replaced `setup-rust-build` in `package-deploy-manifests` job with `dtolnay/rust-toolchain@stable`
145+
146+
### Why
147+
The `firestoned/github-actions/rust/build-binary@v1.3.6` action internally sets `-C linker=x86_64-unknown-linux-gnu-gcc`, which is not installed on GitHub Actions `ubuntu-latest` runners, causing all build jobs to fail. Native `cargo build --release` on arch-appropriate runners eliminates cross-compilation entirely. License-id was stale after the FINOS Apache-2.0 migration.
148+
149+
### Impact
150+
- [ ] Breaking change
151+
- [ ] Requires cluster rollout
152+
- [ ] Config change only
153+
- [x] Documentation only
154+
155+
---
156+
157+
## [2026-04-09 03:00] - Phase 2 (P2-4): leader election via kube-lease-manager
158+
159+
**Author:** Erick Bourgeois
160+
161+
### Changed
162+
- `Cargo.toml`: Added `kube-lease-manager = "0.11"` dependency
163+
- `src/constants.rs`: Added `DEFAULT_LEASE_NAME`, `DEFAULT_LEASE_DURATION_SECS`, `DEFAULT_LEASE_RENEW_DEADLINE_SECS`, `DEFAULT_LEASE_RETRY_PERIOD_SECS`, `DEFAULT_LEASE_GRACE_SECS`, `DEFAULT_LEASE_NAMESPACE` constants
164+
- `src/reconcilers/scheduled_machine.rs`: Added `is_leader: Arc<AtomicBool>` to `Context` (defaults to `true` for backward-compatible single-instance mode); added leader guard in `reconcile_guarded` — non-leaders return `Action::await_change()` immediately
165+
- `src/main.rs`: Added `enable_leader_election`, `lease_name`, `lease_namespace`, `lease_duration_secs`, `lease_renew_deadline_secs` CLI args; when enabled, sets `is_leader = false` at startup and spawns a background `kube-lease-manager` task that flips `is_leader` on acquisition/loss
166+
- `deploy/deployment/deployment.yaml`: Fixed `POD_NAME``CONTROLLER_POD_NAME` env var (aligns with `Context::new` and leader election holder identity)
167+
- `src/reconcilers/scheduled_machine_tests.rs`: Added 2 TDD tests — `test_context_new_defaults_is_leader_to_true` and `test_reconcile_guarded_awaits_change_when_not_leader`
168+
- `docs/src/operations/configuration.md`: Added all leader election env vars, CLI args, Leader Election section, Lease RBAC rules
169+
170+
### Why
171+
Basel III HA (P2-4): a single-replica controller is a single point of failure. With `ENABLE_LEADER_ELECTION=true` and `replicas: 2`, only the lease holder reconciles resources. Standby replicas react within one `LEASE_DURATION_SECONDS` window on leader failure. `Context::is_leader` defaults to `true` so existing single-replica deployments continue without any config change.
172+
173+
### Impact
174+
- [ ] Breaking change
175+
- [x] Requires cluster rollout — set `ENABLE_LEADER_ELECTION=true` and `replicas: 2`; RBAC for `leases` already in `clusterrole.yaml`
176+
- [ ] Config change only
177+
- [ ] Documentation only
178+
179+
---
180+
181+
## [2026-04-09 02:00] - Add SPDX license headers to all .github YAML files
182+
183+
**Author:** Erick Bourgeois
184+
185+
### Changed
186+
- `.github/ISSUE_TEMPLATE/bug_report.yml`: Added `# Copyright (c) 2025 Erick Bourgeois, finos` + `# SPDX-License-Identifier: Apache-2.0` header
187+
- `.github/ISSUE_TEMPLATE/feature_request.yml`: Same
188+
- `.github/ISSUE_TEMPLATE/meeting_minutes.yml`: Same
189+
- `.github/ISSUE_TEMPLATE/support_question.yml`: Same
190+
191+
### Why
192+
Supply-chain provenance and automated license scanning (NIST SA-4) require SPDX headers on all project-owned files. The three workflow files and both composite actions already had headers from P2-10; these four issue templates were the remaining `.github/` YAML files without them. `dco.yml` was intentionally left untouched — it is managed by FINOS and carries an explicit "Do not edit" notice.
193+
194+
### Impact
195+
- [ ] Breaking change
196+
- [ ] Requires cluster rollout
197+
- [ ] Config change only
198+
- [x] Documentation only
199+
200+
---
201+
12202
## [2026-04-09 01:00] - Phase 2 (P2-5): exponential back-off in error policy
13203

14204
**Author:** Erick Bourgeois
@@ -86,7 +276,7 @@ The `ValidatingAdmissionPolicy` deployed in the previous entry had no user-facin
86276
- `src/**/*.rs` (all 18 files): Added P2-10 SPDX supply-chain provenance headers to every Rust source file:
87277
```
88278
// Copyright (c) 2025 Erick Bourgeois, RBC Capital Markets
89-
// SPDX-License-Identifier: MIT
279+
// SPDX-License-Identifier: Apache-2.0
90280
```
91281

92282
### Why

.github/ISSUE_TEMPLATE/bug_report.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
name: 🐛 Bug Report
24
description: If something isn't working as expected 🤔
35
title: "[Bug] "

.github/ISSUE_TEMPLATE/feature_request.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
name: 🚀 Feature Request
24
description: I have a suggestion (and may want to implement it 🙂)!
35
title: "[Feature] "

.github/ISSUE_TEMPLATE/meeting_minutes.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
name: 🐛 Bug Report
24
description: If something isn't working as expected 🤔
35
title: "[Bug] "

.github/ISSUE_TEMPLATE/support_question.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
name: 🤗 Support Question
24
description: If you have a question about configuration, usage, etc. 💬
35
title: "[Support] "

.github/actions/extract-version/action.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Copyright (c) 2025 Erick Bourgeois, finos
2-
# SPDX-License-Identifier: MIT
2+
# SPDX-License-Identifier: Apache-2.0
33

44
name: 'Extract Version Information'
55
description: 'Extracts version, tag name, and image tags for consistent use across workflows'

.github/actions/prepare-docker-binaries/action.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# Copyright (c) 2025 Erick Bourgeois, finos
2-
# SPDX-License-Identifier: MIT
2+
# SPDX-License-Identifier: Apache-2.0
33

44
name: 'Prepare Docker Binaries'
55
description: 'Downloads artifacts and prepares binaries for multi-arch Docker build'

.github/dco.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Copyright (c) 2025 Erick Bourgeois, finos
2+
# SPDX-License-Identifier: Apache-2.0
13
# Do not edit this file! Email help@finos.org if you have any questions.
24

35
allowRemediationCommits:

0 commit comments

Comments
 (0)