User-Defined Node Taints on Provisioned Nodes

[ebourgeois]

Description of Problem

Status: ⏳ Proposed
Author: Erick Bourgeois
Created: 2026-04-20
Scope: CRD schema, reconciler logic, status tracking, RBAC confirmation, docs, examples. No CAPI or bootstrap-provider changes.

Context

When a ScheduledMachine is active, the controller creates CAPI Bootstrap + Infrastructure + Machine resources, CAPI provisions the physical host, it joins the cluster, and a Node object appears. Today the controller tracks that Node via status.nodeRef but never mutates it. Operators who want the Node to carry workload-specific taints (e.g., workload=batch:NoSchedule, dedicated=ml:NoExecute) have to apply them out-of-band — typically via a kubelet --register-with-taints flag in the bootstrap config, or a separate post-join script — which breaks the "one CR defines everything" story for a ScheduledMachine.

We want a first-class field on ScheduledMachineSpec that declares "these taints must exist on the Node once it is Ready," and have the controller apply them idempotently, track what it applied, and reconcile drift.

Goal

Add spec.nodeTaints to ScheduledMachine (a list of {key, value?, effect} entries matching Kubernetes' core/v1 Taint shape). When the Node becomes Ready, the controller patches the Node to add every declared taint, records what it applied in status.appliedNodeTaints, and keeps the declared set and the applied set in sync on every reconcile. Admin-added taints on the same Node are left alone.

Non-goals

• Do not manage tolerations. Tolerations live on Pods (or PodSpec templates), not Nodes. If workloads need to tolerate the taints we apply, that is the workload author's job.
• Do not manage node labels. Labels would be useful but are a separate feature with a different conflict model (labels are key→value; taints are key+effect→value). Track separately.
• Do not remove admin-added taints. Only taints we applied (tracked in status.appliedNodeTaints and via an owner annotation on the Node) may be mutated or removed by the controller.
• Do not re-taint on every reconcile. Read first, diff, patch only if needed, to keep the write-rate low and the audit log clean.
• Do not attempt to taint before the Node exists. No-op until status.nodeRef is populated and the Node is Ready.

Potential Solutions

Current state (snapshot 2026-04-20)

• src/crd.rs:44 — ScheduledMachineSpec has no taint field today.
• src/reconcilers/helpers.rs:1345 — node_to_scheduled_machines pure reverse mapper already reacts to Node changes and enqueues the owning ScheduledMachine. This is exactly the hook we need for drift detection: a manual change to the Node fires a reconcile.
• src/reconcilers/helpers.rs:1379 — cordon_node shows the existing pattern for patching a Node (kube::Api::<Node>::patch with a merge patch). We extend the same pattern for .spec.taints.
• deploy/deployment/rbac/clusterrole.yaml:54-56 — ClusterRole already grants get/list/watch/patch/update on nodes. No RBAC change needed.
• src/crd.rs:329 — status.nodeRef is already populated when CAPI sets Machine.status.nodeRef, so we have a stable hook for "the Node is claimed."
• Event-driven architecture: the .watches() on core Node triggers a reconcile every time the Node's status.conditions[].type=Ready flips or .spec.taints is edited. The roadmap relies on this — no polling.

Phases

Phase 1 — CRD schema + validation (TDD)

• Define NodeTaint in src/crd.rs, mirroring the core/v1 Taint shape:

  #[derive(Clone, Debug, Serialize, Deserialize, JsonSchema, PartialEq, Eq, Hash)]
  #[serde(rename_all = "camelCase")]
  pub struct NodeTaint {
      pub key: String,                 // RFC 1123 label; required
      #[serde(default, skip_serializing_if = "Option::is_none")]
      pub value: Option<String>,       // optional, <=63 chars
      pub effect: TaintEffect,         // enum: NoSchedule | PreferNoSchedule | NoExecute
  }
  
  #[derive(Clone, Debug, Serialize, Deserialize, JsonSchema, PartialEq, Eq, Hash)]
  pub enum TaintEffect { NoSchedule, PreferNoSchedule, NoExecute }

• Add to ScheduledMachineSpec:

  /// User-defined taints applied to the Kubernetes Node once it is Ready.
  /// The controller owns and reconciles only the taints it applied;
  /// admin-added taints on the same Node are left untouched.
  #[serde(default, skip_serializing_if = "Vec::is_empty")]
  pub node_taints: Vec<NodeTaint>,

• Validation (via ValidatingAdmissionPolicy + a Rust-side guard in the reconciler):

  • Reject key that does not match [a-z0-9A-Z]([-a-zA-Z0-9.]*[a-zA-Z0-9])? (same rule Kubernetes' admission uses).
  • Reject duplicate {key, effect} pairs in the same list (core/v1 admission rejects the same duplicate; we match the rule at the CR boundary so users get a clearer error).
  • Reject key prefixed with 5spot.finos.org/ — that's our own reserved namespace.
  • Reject reserved kubelet/system keys (node.kubernetes.io/*, node-role.kubernetes.io/*, kubernetes.io/*) with a clear message pointing users at spec.machineTemplate for any control-plane role signalling.

• Write failing tests first (in src/crd_tests.rs): parse a valid NodeTaint, reject bad effect, reject bad key, reject duplicate {key, effect}, tolerate missing value.

• Run regen-crds skill, then validate-examples skill.

Exit criterion: kubectl apply --dry-run=client -f examples/scheduledmachine-tainted.yaml succeeds for a valid spec and fails with a precise error for each invalid-input test.

Phase 2 — Status surface + applied-taints tracking (TDD)

• Add to ScheduledMachineStatus:

  /// Taints the controller has applied to the Node, in the order they
  /// were applied. Maintained as the controller's record of truth so we
  /// only mutate taints we own.
  #[serde(default, skip_serializing_if = "Vec::is_empty")]
  pub applied_node_taints: Vec<NodeTaint>,

• Add NodeTainted condition type to the condition enum:

  • status=True, reason=Applied: all declared taints present on the Node.
  • status=False, reason=NodeNotReady: Node exists but condition Ready!=True.
  • status=False, reason=PatchFailed: k8s API returned an error on the last patch attempt.
  • status=Unknown, reason=NoNodeYet: status.nodeRef not populated yet.

• src/reconcilers/helpers.rs: add patch_applied_taints_status() following the same pattern as patch_status_machine_refs (helpers.rs:1228). Include observedGeneration.

• Failing tests first: status serialisation round-trip; condition transitions (Unknown → False → True).

Exit criterion: after a successful apply, kubectl get scheduledmachine <name> -o jsonpath='{.status.appliedNodeTaints}' returns the list the controller put there, and the NodeTainted condition flips to True.

Phase 3 — Node-patch helper (pure + isolated, TDD)

Write one pure function that computes the diff, one IO function that performs the patch. The pure function is unit-tested exhaustively; the IO function is thin and integration-tested.

• src/reconcilers/helpers.rs: fn diff_node_taints(current: &[Taint], desired: &[NodeTaint], previously_applied: &[NodeTaint]) -> NodeTaintPlan

  Where NodeTaintPlan is:

  pub struct NodeTaintPlan {
      pub to_add: Vec<NodeTaint>,     // in desired, not on current
      pub to_update: Vec<NodeTaint>,  // on current with same (key,effect) but different value
      pub to_remove: Vec<NodeTaint>,  // in previously_applied, not in desired; leave admin taints alone
      pub unchanged: Vec<NodeTaint>,  // no-op
  }

  Semantics (non-obvious parts):

  • A taint is identified by the tuple (key, effect) — value is mutable. This matches how core/v1 admission treats taints.
  • to_remove is bounded by previously_applied ∩ current. If the admin has independently added a taint whose (key, effect) collides with ours, we do not remove it — log a warning and surface a TaintOwnershipConflict condition instead.
  • An empty plan returns unchanged populated and to_add/to_update/to_remove empty — the caller uses this to skip the patch round-trip.

• async fn apply_node_taints(client: &Client, node_name: &str, plan: &NodeTaintPlan) -> Result<(), ReconcilerError>

  • Server-side apply with field_manager: "5spot-controller" so our ownership is explicit in the Node's managedFields.
  • Uses a merge patch rebuilding .spec.taints from the current Node's list minus to_remove plus to_add/to_update.
  • Also writes the annotation 5spot.finos.org/applied-taints: <json-array-of-keys-we-own> onto the Node so a second controller (or a human) can see our ownership at a glance without reading the ScheduledMachine CR.

• Tests for diff_node_taints: 100% branch coverage — empty/empty, add-only, remove-only, update-only, mixed, ownership conflict, deduplicate-admin-collision.

• Tests for apply_node_taints via kube::test or a mocked client.

Exit criterion: cargo test --lib reconcilers::helpers::tests::diff_node_taints passes with 100% branch coverage; clippy clean.

Phase 4 — Wire into the reconciler

• In src/reconcilers/scheduled_machine.rs, add a new phase-handler call in the "Active" branch after status.nodeRef is populated:

  if let Some(node_ref) = &sm.status.as_ref().and_then(|s| s.node_ref.clone()) {
      reconcile_node_taints(&ctx.client, &sm, node_ref).await?;
  }

• reconcile_node_taints:

  1. Fetch the Node (kube::Api::<Node>::get(name)). If 404, emit NodeTainted=Unknown/NoNodeYet and return with short requeue (Node is likely about to appear).
  2. Check Node Ready condition. If not Ready, emit NodeTainted=False/NodeNotReady and return with short requeue.
  3. Call diff_node_taints. If plan is fully unchanged, return Ok.
  4. Call apply_node_taints. On error, increment applyFailureCount condition field, requeue with exponential backoff (reuse the existing ERROR_REQUEUE_SECS constant).
  5. On success, patch ScheduledMachineStatus.appliedNodeTaints to match the new desired list.
  6. Emit event: NodeTaintsApplied key1=v1:NoSchedule, key2:NoExecute.

• Requeue timing: no proactive requeue needed on success — the Node watch is authoritative for drift detection. On transient failure use the existing backoff policy.

• Remove-taints path: when sm.status.appliedNodeTaints is non-empty but the spec list is now empty, the diff will produce to_remove and the apply will patch them out in one round-trip.

Exit criterion: end-to-end flow on a kind cluster: create a ScheduledMachine with spec.nodeTaints, watch a Node join, observe the taints appear on the Node and the NodeTainted condition flip to True within one reconcile cycle.

Phase 5 — Termination and cleanup

When a ScheduledMachine deactivates (schedule ends, killSwitch, or deletion), the existing drain + Machine-delete flow already removes the Node from the cluster, so explicit taint removal is unnecessary. Two edge cases still need handling:

• killSwitch with NoExecute taint declared but not yet applied: if the spec includes a NoExecute taint and the kill switch fires before the Node was Ready, we skip the taint step entirely. Do not try to apply taints as a shortcut for eviction — drain is the sanctioned path.
• ScheduledMachine is deleted while the Node still exists (CAPI slow-delete): the finalizer already waits for Machine deletion before the ScheduledMachine is GC'd. During that window, if the user deletes the SM object, we stop reconciling taints — the Machine-delete flow will tear the Node down on its own cadence.
• Unit test: deleting a ScheduledMachine mid-apply does not leave the Node in a partially-tainted state (the node is going away anyway).

Exit criterion: no dangling-taint scenarios — every code path that ends a ScheduledMachine's life either deletes the Node (drain + Machine delete, so taints follow) or stops reconciling (deletion during GC window).

Phase 6 — ValidatingAdmissionPolicy (defense in depth)

• Extend deploy/admission/validatingadmissionpolicy.yaml to reject at admission time:
  • Duplicate (key, effect) in spec.nodeTaints
  • Taint key prefixed with 5spot.finos.org/ or any reserved kubernetes.io namespace
  • Taint key or value longer than 63 chars
• These rules are duplicated between the Rust-side validator and the VAP for two reasons: (1) the VAP stops bad input at API-server admission before the reconciler sees it, saving a reconcile cycle, and (2) the Rust-side guard is the safety net for clusters without the VAP installed.
• Unit test the VAP CEL expressions.

Exit criterion: kubectl apply -f examples/scheduledmachine-bad-taint.yaml is rejected by the API server with the VAP's CEL error message, before the reconciler queues anything.

Phase 7 — Docs, examples, changelog

• docs/src/concepts/scheduled-machine.md: add a "Node Taints" subsection explaining the field, the ownership model (only our taints are managed), and pointing at the examples.
• examples/scheduledmachine-tainted.yaml: a worked example with two taints (one NoSchedule, one NoExecute).
• docs/src/reference/api.md: regenerated via make crddoc last, per the CLAUDE.md order-of-operations rule.
• docs/src/operations/troubleshooting.md: new entry "Taints not appearing on Node" walking through the three NodeTainted condition states and how to debug each.
• README.md: one-line mention in the features list.
• .claude/CHANGELOG.md: per-phase entries with Author attribution.

Exit criterion: sync-docs skill reports no drift; make crddoc output matches checked-in docs/src/reference/api.md.

Phase 8 — Integration test against a real kind cluster

• Add tests/integration_node_taints.rs:
  • Spin up kind with the CRD + controller image (reusing existing test harness).
  • Apply a ScheduledMachine with two taints.
  • Wait for the controller to create the CAPI Machine (or mock it to populate status.nodeRef directly).
  • Assert the Node gets both taints, the annotation, and the status.
  • Mutate the spec to remove one taint; assert the controller removes it from the Node, keeps the other, and updates status.
  • Externally add a taint via kubectl taint; reconcile; assert the controller does not remove it and surfaces TaintOwnershipConflict=False rather than NodeTainted=True.

Exit criterion: CI integration job for taints passes; no flakes across 10 consecutive runs.

Risks and mitigations

Risk	Mitigation
Controller removes admin-added taints that collide with a declared one	previously_applied list + annotation on the Node are the source of truth for what we own. to_remove is bounded by that intersection. Collision case surfaces as TaintOwnershipConflict condition, never as a silent overwrite.
Applying NoExecute evicts workloads the operator did not expect to reschedule	Document the effect clearly in the user guide. Consider a PR-gate lint: NoExecute taints should have a value matching a known policy (e.g., =dedicated:NoExecute). Do not block in code — this is operator policy, not controller policy.
Race between CAPI populating status.nodeRef and our first taint attempt	Phase 4 step 1 handles the 404: short-requeue, no error. Node watch fires the reconcile again as soon as the Node exists. No polling.
Patch conflicts with another controller applying taints via server-side apply	Use field_manager: "5spot-controller" on every apply. kube-rs surfaces conflict errors clearly; escalate as a TaintOwnershipConflict condition and stop trying until the spec changes.
spec.nodeTaints mutated while the reconciler is mid-apply	Reconcile is ~100ms total; next reconcile picks up the new spec. No locking needed. Status will briefly show the old applied list until the next reconcile succeeds, which is accurate.
User sets NoExecute on a Node that is a CAPI control plane	Reject at admission (Phase 6) — CAPI control-plane Nodes are not created by ScheduledMachine today, but VAP is the cheap insurance policy.
Node watch fires a reconcile for every unrelated taint change elsewhere in the cluster	Existing reverse-mapper node_to_scheduled_machines already filters by matching Node name against the Controller's own Store (helpers.rs:1345). An unrelated Node taint change results in an empty Vec enqueue and costs nothing.

Verification

At each phase boundary:

• cargo-quality skill (fmt + clippy + test) — 100% unit test coverage per the TDD rule.
• validate-examples skill — all example YAMLs apply cleanly.
• sync-docs skill — no doc/code drift.

End-to-end (Phase 8):

• Integration test green for 10 consecutive CI runs.
• kubectl describe node <name> shows the expected taints and the 5spot.finos.org/applied-taints annotation.
• kubectl get scheduledmachine <name> -o jsonpath='{.status.appliedNodeTaints}' matches spec.nodeTaints.

Sequencing and landing strategy

Phases 1–3 are internal (schema + pure helpers + status plumbing) and can land in one PR. Phase 4 (wiring) is a second PR — it turns the feature on. Phase 5 (termination edges) is a third PR; Phase 6 (VAP) is a fourth (lives in deploy/admission/ so it can go in independently). Phases 7–8 (docs + integration tests) land last, one PR each.

Each PR is TDD-first, accompanies a CHANGELOG entry with Author attribution, passes the pre-commit-checklist skill, and regenerates CRDs + API docs per the mandatory CRD order-of-operations.

Out of scope — explicit list

• Node labels management (separate roadmap)
• Tolerations on Pods (workload concern, not infrastructure)
• CAPI Machine.spec.taints field (that's for CAPI's own node-registration flow; we are the post-join layer)
• Taints applied via the bootstrap provider's kubelet-extra-args (operators can still do this; it is additive to our feature, not a competitor)
• Node annotations beyond the one ownership annotation we need

[ebourgeois]

Implemented in: #41