feat: remove node startup taint on driver start - #2052
Conversation
|
This issue is currently awaiting triage. If secrets-store-csi-driver contributors determine this is a relevant issue, they will accept it by applying the The DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
Welcome @aaresh-sharma! |
|
Hi @aaresh-sharma. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Regular contributors should join the org to skip this step. Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
|
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: aaresh-sharma The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Pods that mount SecretProviderClass volumes can be scheduled onto a node before the secrets-store CSI driver pod is running there, and fail to mount until it is. Cluster operators can now close this race by tainting nodes with secrets-store.csi.k8s.io/agent-not-ready at registration (for example via EKS Managed Node Group taints). The driver removes that taint from its own node once it starts, so workload pods only schedule after the driver is ready. The mechanism follows the aws-fsx-csi-driver implementation: a background goroutine reads the node name from KUBE_NODE_NAME, gets the Node, and patches out any taint with the matching key, retrying with exponential backoff. The JSON patch pairs a test op on /spec/taints with the replace so a concurrent taint update rejects the patch and the retry re-reads fresh state. On clusters that never apply the taint, or when KUBE_NODE_NAME is unset, this is a no-op. RBAC adds nodes get/patch to the ClusterRole (staging chart and kustomize; released copies promoted at release time).
78a89b7 to
2360eac
Compare
|
/easycla |
|
/assign @aramase |
What type of PR is this?
/kind feature
What this PR does / why we need it:
Adds node startup taint removal, the same pattern the AWS FSx, EFS, and EBS CSI drivers ship.
There is a race on node startup: pods that mount
SecretProviderClassvolumes can be scheduled onto a new node before the driver pod is running there, and their mounts fail until it is. Operators can now close this race by tainting nodes at registration withsecrets-store.csi.k8s.io/agent-not-ready(EKS Managed Node Groups, for example, can apply taints automatically). The driver's node pod tolerates all taints, so it still schedules, and once it starts it removes the taint so workload pods can follow.How it works:
mainlaunches a background goroutine at startup (RemoveNotReadyTaintInBackground).KUBE_NODE_NAMEenv var (already injected viafieldRef: spec.nodeNamein all four DaemonSets), gets the Node, and removes any taint whose key matches, retrying with exponential backoff. Driver startup is never blocked on this.testop on/spec/taintswith thereplace, so a concurrent change to the node's taints rejects the patch and the retry re-reads fresh state. This is the one intentional difference from the FSx implementation, which patches without the guard.KUBE_NODE_NAMEis unset, this is a no-op.RBAC: the ClusterRole needs
getandpatchonnodes. Added via kubebuilder marker; staging chart and kustomize output regenerated. Released chart and deploy copies are left alone for promotion at release time. No new configurable values, so no chart configuration table changes.Docs: new "Configure node startup taint" section in the installation guide.
Testing:
[], notnull), patch rejection on concurrent modification, and backoff retry.NO_EXECUTE) in its config. Nodes register already tainted, a pinned workload stays Pending while the taint is live, and it binds in the same second the driver removes the taint. Removal also worked over the cluster's IPv6 in-cluster API path.Which issue(s) this PR fixes:
None in this repo. Motivated by aws/secrets-store-csi-driver-provider-aws#454, which asks for exactly this. That request was filed against the AWS provider, but the node service that gates readiness lives here in the driver, and the fix benefits every provider.
Special notes for your reviewer:
nodesRBAC marker sits with the other ClusterRole markers on the SecretProviderClassPodStatus reconciler, with a comment noting the actual consumer ispkg/secrets-store/node_taint.go.KUBE_NODE_NAMEis present in the Windows DaemonSets, but I have only validated on Linux nodes.TODOs:
🤖 Generated with Claude Code