Skip to content

Commit 043c13d

Browse files
ebourgeoisebourgeois
committed
Emergency Node Reclaim by Process Match
Signed-off-by: Erick Bourgeois <erick@jeb.ca>
1 parent 04e2306 commit 043c13d

29 files changed

Lines changed: 4416 additions & 24 deletions

‎.claude/CHANGELOG.md‎

Lines changed: 297 additions & 0 deletions
Large diffs are not rendered by default.

‎.trivyignore‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,105 @@ AVD-KSV-0125
6363
# ConfigMaps — all sensitive material is sourced from Secrets.
6464
AVD-KSV-01010
6565

66+
# ─────────────────────────────────────────────────────────────────────────────
67+
# DaemonSet — deploy/node-agent/daemonset.yaml
68+
# ─────────────────────────────────────────────────────────────────────────────
69+
70+
# KSV-0010 — Access to host PID namespace (hostPID: true).
71+
# Architecturally required. The reclaim-agent is a host-process watcher: it
72+
# detects the emergency-reclaim match by scanning /proc for processes whose
73+
# argv matches a curated kill-list, and it executes the graceful stop by
74+
# signalling those host PIDs. Both capabilities depend on the agent sharing
75+
# the host's PID namespace — pod-scoped /proc would only expose the agent's
76+
# own container and make the entire feature inoperable. The agent runs with
77+
# a narrow SecurityContext (read-only root fs, no privilege escalation, no
78+
# extra capabilities beyond what hostPID implies) and is deployed only on
79+
# nodes that have opted in via the 5spot.finos.org/reclaim-agent label.
80+
AVD-KSV-0010
81+
82+
# KSV-0121 — DaemonSet shouldn't have volumes set to {"/proc"}.
83+
# Architecturally required (HIGH-severity variant of KSV-0023 that names
84+
# the specific disallowed host paths). Same rationale: the agent's
85+
# detection contract IS reading `/proc/<pid>/{comm,cmdline}` for every
86+
# host process, and Kubernetes exposes no non-hostPath mechanism for
87+
# that. The mount is scoped to exactly `/proc`, `readOnly: true`,
88+
# `type: Directory`, and the agent never writes back. Phase 2 rung-2
89+
# (netlink proc connector) will replace the read-side but still needs
90+
# host `/proc` for the initial match fan-out, so this suppression is
91+
# durable. No other hostPath volumes exist on this DaemonSet.
92+
AVD-KSV-0121
93+
94+
# KSV-0023 — hostPath volume mounted.
95+
# Architecturally required. The single hostPath volume is `/proc`, mounted
96+
# read-only at /host/proc so the agent can read /proc/<pid>/{comm,cmdline}
97+
# for every host process. The detection contract IS reading host process
98+
# state — there is no Kubernetes-native substitute (the Downward API,
99+
# projected volumes, and CSI drivers do not expose kernel /proc). Scope is
100+
# already minimal: one path (`/proc`), read-only, `type: Directory`, and
101+
# the agent writes nothing back. No other hostPath volumes exist in the
102+
# DaemonSet. Future rung-2 work (netlink proc connector) will replace the
103+
# /proc read-side with event-driven push but still requires host /proc for
104+
# the initial match fan-out, so this suppression is durable.
105+
AVD-KSV-0023
106+
107+
# KSV-0020 — Container 'agent' securityContext.runAsUser should be > 10000.
108+
# Architecturally required (same root cause as KSV-0012 / KSV-0105 /
109+
# KSV-0118 — this rule is the "high-UID" strictness variant of KSV-0105).
110+
# UID 0 is required for `/proc/<pid>/{comm,cmdline}` reads under hardened
111+
# `hidepid=2` mounts. The rule's intent (avoid low-numbered UIDs that may
112+
# collide with reserved host accounts) is moot at UID 0: the collision
113+
# with `root` is deliberate, not accidental. Every other hardening lever
114+
# is already pulled (readOnlyRootFilesystem, capabilities.drop [ALL],
115+
# no privilege escalation, seccomp RuntimeDefault, opt-in nodeSelector).
116+
AVD-KSV-0020
117+
118+
# KSV-0105 — Container 'agent' securityContext.runAsUser should be > 0.
119+
# Architecturally required (same root cause as KSV-0012 / KSV-0118 —
120+
# different rule wording, identical finding). UID 0 is required to read
121+
# `/proc/<pid>/{comm,cmdline}` under hardened `hidepid=2` /proc mounts;
122+
# the only non-root alternative is re-adding `CAP_SYS_PTRACE`, which is
123+
# strictly broader. Every other hardening lever is already pulled:
124+
# readOnlyRootFilesystem, capabilities.drop [ALL], no privilege escalation,
125+
# seccomp RuntimeDefault, opt-in nodeSelector.
126+
AVD-KSV-0105
127+
128+
# KSV-0021 — Container 'agent' should set securityContext.runAsGroup > 10000.
129+
# Architecturally required (follows from KSV-0012 / KSV-0118). The agent
130+
# runs as UID 0 to read host /proc under `hidepid=2`; UID 0 implies GID 0,
131+
# so runAsGroup > 10000 cannot be satisfied without abandoning the root
132+
# requirement already justified above. The rule exists to avoid a shared
133+
# high-GID supplementary group leaking filesystem access between
134+
# unrelated workloads — not a risk here, since the container mounts only
135+
# host /proc (read-only) and has `readOnlyRootFilesystem: true` with
136+
# `capabilities.drop: [ALL]`.
137+
AVD-KSV-0021
138+
139+
# KSV-0012 — Container 'agent' should set securityContext.runAsNonRoot: true.
140+
# Architecturally required (same root cause as KSV-0118 — this rule is the
141+
# container-level variant, 0118 is the pod-level variant). Running as UID 0
142+
# is required so the agent can read /proc/<pid>/{comm,cmdline} for every
143+
# host PID under hardened `hidepid=2` /proc mounts. A non-root build would
144+
# need CAP_SYS_PTRACE added back, which is a strictly broader privilege
145+
# than running as root with every other capability dropped, the filesystem
146+
# read-only, `allowPrivilegeEscalation: false`, and seccomp RuntimeDefault.
147+
# The agent is also scoped to opt-in nodes via the
148+
# `5spot.finos.org/reclaim-agent: enabled` nodeSelector, so even the
149+
# root-process surface is bounded by the operator's explicit arming.
150+
AVD-KSV-0012
151+
152+
# KSV-0118 — DaemonSet uses the default security context (root allowed).
153+
# False-positive wording — an explicit pod- and container-level
154+
# SecurityContext IS set (runAsUser: 0, runAsGroup: 0, runAsNonRoot: false,
155+
# seccompProfile: RuntimeDefault, allowPrivilegeEscalation: false,
156+
# readOnlyRootFilesystem: true, capabilities.drop: [ALL]). The rule fires
157+
# because runAsNonRoot is false, not because the context is missing.
158+
# Running as UID 0 is architecturally required: reading
159+
# /proc/<pid>/{comm,cmdline} for every host PID needs root under hardened
160+
# hidepid=2 /proc mounts. A non-root variant would require adding
161+
# CAP_SYS_PTRACE back, which is a broader privilege than running as root
162+
# with every other capability dropped and the filesystem read-only.
163+
AVD-KSV-0118
164+
66165
# ─────────────────────────────────────────────────────────────────────────────
67166
# Dockerfile — Dockerfile, Dockerfile.chainguard
68167
# ─────────────────────────────────────────────────────────────────────────────

‎Cargo.lock‎

Lines changed: 60 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,11 @@ tower = "0.5"
4646
regex = "1"
4747
lazy_static = "1.5"
4848
warp = "0.3"
49+
toml = "0.8"
50+
51+
[[bin]]
52+
name = "5spot-reclaim-agent"
53+
path = "src/bin/reclaim_agent.rs"
4954

5055
[dev-dependencies]
5156
tokio-test = "0.4"

‎Makefile‎

Lines changed: 163 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
# Copyright (c) 2025 Erick Bourgeois, RBC Capital Markets
22
# SPDX-License-Identifier: Apache-2.0
33

4-
.PHONY: help install build build-debug build-linux-amd64 build-linux-arm64 build-macos-arm64 prepare-binaries-linux-amd64 prepare-binaries-linux-arm64 test test-lib lint format clean crds crddoc docs docs-serve docs-clean docs-rustdoc calm-diagrams calm-validate run-local docker-build docker-build-amd64 docker-build-arm64 docker-build-chainguard docker-push docker-buildx docker-buildx-chainguard gitleaks gitleaks-install install-git-hooks security-scan-local sbom audit
4+
.PHONY: help install build build-debug build-linux-amd64 build-linux-arm64 build-macos-arm64 prepare-binaries-linux-amd64 prepare-binaries-linux-arm64 test test-lib lint format clean crds crddoc docs docs-serve docs-clean docs-rustdoc calm-diagrams calm-validate run-local docker-build docker-build-amd64 docker-build-arm64 docker-build-chainguard docker-push docker-buildx docker-buildx-chainguard gitleaks gitleaks-install install-git-hooks security-scan-local sbom audit kind-install kind-create kind-delete kind-load kind-deploy kind-example kind-setup kind-status
55

66
# CALM (FINOS Common Architecture Language Model) configuration
77
CALM_CLI_VERSION ?= 1.37.0
@@ -37,6 +37,17 @@ CONTAINER_TOOL ?= docker
3737
# Security tool versions
3838
GITLEAKS_VERSION ?= 8.21.2
3939

40+
# Kind (local Kubernetes) configuration
41+
KIND_VERSION ?= 0.24.0
42+
KIND_CLUSTER_NAME ?= 5spot-dev
43+
KIND_NODE_IMAGE ?= kindest/node:v1.31.0
44+
# Image reference used for the locally built kind image. The `local-dev` tag
45+
# makes it unambiguous that the image is a developer build (not a released
46+
# version). `kind-deploy` applies the checked-in Deployment (pinned to
47+
# ghcr.io/finos/5-spot:v0.1.0) and then overrides the container image to
48+
# $(KIND_IMAGE) via `kubectl set image` so the locally loaded image is used.
49+
KIND_IMAGE ?= ghcr.io/finos/5-spot:local-dev
50+
4051
# Python/Poetry package index configuration (for corporate environments)
4152
# Set PYPI_INDEX_URL to use a custom PyPI mirror (e.g., Artifactory)
4253
# Example: export PYPI_INDEX_URL=https://artifactory.example.com/api/pypi/pypi/simple
@@ -437,3 +448,154 @@ sbom: ## Generate CycloneDX SBOM (Software Bill of Materials)
437448
audit: ## Check dependencies for security vulnerabilities (installs cargo-audit if missing)
438449
@command -v cargo-audit >/dev/null 2>&1 || { echo "Installing cargo-audit..."; cargo install cargo-audit; }
439450
@cargo audit
451+
452+
# ============================================================
453+
# Kind Cluster (local testing for ScheduledMachine)
454+
# ============================================================
455+
456+
kind-install: ## Install kind CLI if missing (verifies checksum)
457+
@if ! command -v kind >/dev/null 2>&1; then \
458+
echo "Installing kind v$(KIND_VERSION)..."; \
459+
OS=$$(uname -s | tr '[:upper:]' '[:lower:]'); \
460+
ARCH=$$(uname -m); \
461+
case "$$ARCH" in \
462+
x86_64) ARCH="amd64" ;; \
463+
aarch64|arm64) ARCH="arm64" ;; \
464+
esac; \
465+
BIN="kind-$${OS}-$${ARCH}"; \
466+
BASE_URL="https://github.com/kubernetes-sigs/kind/releases/download/v$(KIND_VERSION)"; \
467+
echo "Downloading $$BIN..."; \
468+
curl -sSLf -o /tmp/$$BIN "$$BASE_URL/$$BIN"; \
469+
echo "Downloading checksum..."; \
470+
curl -sSLf -o /tmp/$$BIN.sha256sum "$$BASE_URL/$$BIN.sha256sum"; \
471+
echo "Verifying checksum..."; \
472+
cd /tmp && \
473+
EXPECTED=$$(awk '{print $$1}' $$BIN.sha256sum) && \
474+
if command -v sha256sum >/dev/null 2>&1; then \
475+
ACTUAL=$$(sha256sum $$BIN | awk '{print $$1}'); \
476+
else \
477+
ACTUAL=$$(shasum -a 256 $$BIN | awk '{print $$1}'); \
478+
fi && \
479+
if [ "$$EXPECTED" != "$$ACTUAL" ]; then \
480+
echo "ERROR: checksum mismatch (expected $$EXPECTED, got $$ACTUAL)"; \
481+
rm -f /tmp/$$BIN /tmp/$$BIN.sha256sum; \
482+
exit 1; \
483+
fi; \
484+
chmod +x /tmp/$$BIN; \
485+
sudo mv /tmp/$$BIN /usr/local/bin/kind; \
486+
rm -f /tmp/$$BIN.sha256sum; \
487+
echo "✓ kind v$(KIND_VERSION) installed"; \
488+
else \
489+
echo "✓ kind already installed: $$(kind version)"; \
490+
fi
491+
@command -v kubectl >/dev/null 2>&1 || { echo "ERROR: kubectl not found on PATH. Install kubectl and retry."; exit 1; }
492+
493+
kind-create: kind-install ## Create local kind cluster for testing ScheduledMachine
494+
@if kind get clusters 2>/dev/null | grep -qx $(KIND_CLUSTER_NAME); then \
495+
echo "✓ kind cluster '$(KIND_CLUSTER_NAME)' already exists"; \
496+
else \
497+
echo "Creating kind cluster '$(KIND_CLUSTER_NAME)' using $(KIND_NODE_IMAGE)..."; \
498+
kind create cluster --name $(KIND_CLUSTER_NAME) --image $(KIND_NODE_IMAGE) --wait 120s; \
499+
echo "✓ cluster '$(KIND_CLUSTER_NAME)' ready"; \
500+
fi
501+
@kubectl --context kind-$(KIND_CLUSTER_NAME) cluster-info
502+
503+
kind-delete: ## Delete the local kind cluster
504+
@if kind get clusters 2>/dev/null | grep -qx $(KIND_CLUSTER_NAME); then \
505+
kind delete cluster --name $(KIND_CLUSTER_NAME); \
506+
echo "✓ cluster '$(KIND_CLUSTER_NAME)' deleted"; \
507+
else \
508+
echo "✓ no cluster named '$(KIND_CLUSTER_NAME)' — nothing to delete"; \
509+
fi
510+
511+
kind-load: ## Build image for host arch (native cross-compile, bypassing cross) and load it into the kind cluster
512+
@if ! kind get clusters 2>/dev/null | grep -qx $(KIND_CLUSTER_NAME); then \
513+
echo "ERROR: kind cluster '$(KIND_CLUSTER_NAME)' does not exist. Run: make kind-create"; \
514+
exit 1; \
515+
fi
516+
@HOST_ARCH=$$(uname -m); \
517+
case "$$HOST_ARCH" in \
518+
arm64|aarch64) TRIPLE=aarch64-unknown-linux-gnu; DOCKER_ARCH=arm64; LINKER=aarch64-unknown-linux-gnu-gcc ;; \
519+
x86_64|amd64) TRIPLE=x86_64-unknown-linux-gnu; DOCKER_ARCH=amd64; LINKER=x86_64-unknown-linux-gnu-gcc ;; \
520+
*) echo "ERROR: unsupported host arch: $$HOST_ARCH"; exit 1 ;; \
521+
esac; \
522+
echo "Host $$HOST_ARCH -> building linux/$$DOCKER_ARCH image"; \
523+
if ! command -v $$LINKER >/dev/null 2>&1; then \
524+
echo "ERROR: linker '$$LINKER' not found on PATH."; \
525+
echo " On macOS: brew tap messense/macos-cross-toolchains && brew install $$TRIPLE"; \
526+
echo " On Linux: install the matching gcc cross toolchain for your distro."; \
527+
exit 1; \
528+
fi; \
529+
if ! rustup target list --installed | grep -qx "$$TRIPLE"; then \
530+
echo "Adding rustup target $$TRIPLE..."; \
531+
rustup target add "$$TRIPLE"; \
532+
fi; \
533+
echo "Compiling 5spot for $$TRIPLE..."; \
534+
cargo build --release --target "$$TRIPLE"; \
535+
echo "Staging binary at binaries/$$DOCKER_ARCH/5spot..."; \
536+
mkdir -p "binaries/$$DOCKER_ARCH"; \
537+
cp "target/$$TRIPLE/release/5spot" "binaries/$$DOCKER_ARCH/5spot"; \
538+
echo "Building docker image $(KIND_IMAGE) (linux/$$DOCKER_ARCH)..."; \
539+
$(CONTAINER_TOOL) build \
540+
--build-arg TARGETARCH=$$DOCKER_ARCH \
541+
--build-arg VERSION="$(VERSION)" \
542+
--build-arg GIT_SHA="$(GIT_SHA)" \
543+
--build-arg BASE_IMAGE="$(BASE_IMAGE)" \
544+
-t $(KIND_IMAGE) .; \
545+
echo "Loading $(KIND_IMAGE) into kind cluster '$(KIND_CLUSTER_NAME)'..."; \
546+
kind load docker-image $(KIND_IMAGE) --name $(KIND_CLUSTER_NAME); \
547+
echo "✓ image loaded"
548+
549+
kind-deploy: ## Apply CRDs and controller manifests to the kind cluster
550+
@if ! kind get clusters 2>/dev/null | grep -qx $(KIND_CLUSTER_NAME); then \
551+
echo "ERROR: kind cluster '$(KIND_CLUSTER_NAME)' does not exist. Run: make kind-create"; \
552+
exit 1; \
553+
fi
554+
@echo "Applying CRDs to kind cluster '$(KIND_CLUSTER_NAME)'..."
555+
@kubectl --context kind-$(KIND_CLUSTER_NAME) apply -f deploy/crds/
556+
@echo "Applying namespace first (avoids race with namespace-scoped resources)..."
557+
@kubectl --context kind-$(KIND_CLUSTER_NAME) apply -f deploy/deployment/namespace.yaml
558+
@for i in 1 2 3 4 5 6 7 8 9 10; do \
559+
if kubectl --context kind-$(KIND_CLUSTER_NAME) get namespace 5spot-system >/dev/null 2>&1; then \
560+
break; \
561+
fi; \
562+
echo " waiting for namespace 5spot-system ($$i/10)..."; \
563+
sleep 1; \
564+
done
565+
@echo "Applying controller manifests (rbac, configmap, deployment, service, ...)..."
566+
@kubectl --context kind-$(KIND_CLUSTER_NAME) apply -R -f deploy/deployment/
567+
@echo "Overriding controller image to $(KIND_IMAGE) (locally built)..."
568+
@kubectl --context kind-$(KIND_CLUSTER_NAME) -n 5spot-system set image deployment/5spot-controller controller=$(KIND_IMAGE)
569+
@echo "Waiting for the controller Deployment to become available..."
570+
@kubectl --context kind-$(KIND_CLUSTER_NAME) -n 5spot-system rollout status deployment/5spot-controller --timeout=180s
571+
@echo "✓ controller deployed"
572+
573+
kind-example: ## Apply the basic ScheduledMachine example to the kind cluster
574+
@if ! kind get clusters 2>/dev/null | grep -qx $(KIND_CLUSTER_NAME); then \
575+
echo "ERROR: kind cluster '$(KIND_CLUSTER_NAME)' does not exist. Run: make kind-create"; \
576+
exit 1; \
577+
fi
578+
@kubectl --context kind-$(KIND_CLUSTER_NAME) apply -f examples/scheduledmachine-basic.yaml
579+
@echo "✓ example ScheduledMachine applied"
580+
@echo "Inspect with:"
581+
@echo " kubectl --context kind-$(KIND_CLUSTER_NAME) get scheduledmachines -A"
582+
@echo " kubectl --context kind-$(KIND_CLUSTER_NAME) describe scheduledmachine business-hours-worker"
583+
584+
kind-status: ## Show kind cluster, controller, and ScheduledMachine status
585+
@echo "=== kind clusters ==="
586+
@kind get clusters 2>/dev/null || echo "(none)"
587+
@echo ""
588+
@echo "=== controller pods (namespace 5spot-system) ==="
589+
@kubectl --context kind-$(KIND_CLUSTER_NAME) -n 5spot-system get pods 2>/dev/null || echo "(cluster unreachable)"
590+
@echo ""
591+
@echo "=== ScheduledMachines (all namespaces) ==="
592+
@kubectl --context kind-$(KIND_CLUSTER_NAME) get scheduledmachines -A 2>/dev/null || echo "(cluster unreachable)"
593+
594+
kind-setup: kind-create kind-load kind-deploy ## One-shot: create cluster, build+load image, deploy CRDs & controller
595+
@echo ""
596+
@echo "✓ kind setup complete"
597+
@echo ""
598+
@echo "Next steps:"
599+
@echo " make kind-example # apply the example ScheduledMachine"
600+
@echo " make kind-status # see cluster & controller state"
601+
@echo " make kind-delete # tear down"

0 commit comments

Comments
 (0)