Skip to content

feat: remove node startup taint on driver start - #2052

Open
aaresh-sharma wants to merge 1 commit into
kubernetes-sigs:mainfrom
aaresh-sharma:node-startup-taint
Open

feat: remove node startup taint on driver start#2052
aaresh-sharma wants to merge 1 commit into
kubernetes-sigs:mainfrom
aaresh-sharma:node-startup-taint

Conversation

@aaresh-sharma

Copy link
Copy Markdown

What type of PR is this?
/kind feature

What this PR does / why we need it:

Adds node startup taint removal, the same pattern the AWS FSx, EFS, and EBS CSI drivers ship.

There is a race on node startup: pods that mount SecretProviderClass volumes can be scheduled onto a new node before the driver pod is running there, and their mounts fail until it is. Operators can now close this race by tainting nodes at registration with secrets-store.csi.k8s.io/agent-not-ready (EKS Managed Node Groups, for example, can apply taints automatically). The driver's node pod tolerates all taints, so it still schedules, and once it starts it removes the taint so workload pods can follow.

How it works:

  • main launches a background goroutine at startup (RemoveNotReadyTaintInBackground).
  • The goroutine reads the node name from the existing KUBE_NODE_NAME env var (already injected via fieldRef: spec.nodeName in all four DaemonSets), gets the Node, and removes any taint whose key matches, retrying with exponential backoff. Driver startup is never blocked on this.
  • The JSON patch pairs an RFC 6902 test op on /spec/taints with the replace, so a concurrent change to the node's taints rejects the patch and the retry re-reads fresh state. This is the one intentional difference from the FSx implementation, which patches without the guard.
  • On clusters that never apply the taint, or when KUBE_NODE_NAME is unset, this is a no-op.

RBAC: the ClusterRole needs get and patch on nodes. Added via kubebuilder marker; staging chart and kustomize output regenerated. Released chart and deploy copies are left alone for promotion at release time. No new configurable values, so no chart configuration table changes.

Docs: new "Configure node startup taint" section in the installation guide.

Testing:

  • Unit tests cover the no-op paths, selective removal (unrelated taints are kept), sole-taint serialization ([], not null), patch rejection on concurrent modification, and backoff retry.
  • Validated on EKS 1.34 (AL2023, dual-stack): 7/7 e2e checks against pre-tainted nodes, and 8/8 against a managed nodegroup created with the taint (NO_EXECUTE) in its config. Nodes register already tainted, a pinned workload stays Pending while the taint is live, and it binds in the same second the driver removes the taint. Removal also worked over the cluster's IPv6 in-cluster API path.

Which issue(s) this PR fixes:

None in this repo. Motivated by aws/secrets-store-csi-driver-provider-aws#454, which asks for exactly this. That request was filed against the AWS provider, but the node service that gates readiness lives here in the driver, and the fix benefits every provider.

Special notes for your reviewer:

  • The nodes RBAC marker sits with the other ClusterRole markers on the SecretProviderClassPodStatus reconciler, with a comment noting the actual consumer is pkg/secrets-store/node_taint.go.
  • The code compiles into the Windows build and KUBE_NODE_NAME is present in the Windows DaemonSets, but I have only validated on Linux nodes.
  • Behavior is always on and a no-op without the taint. If you would rather have an opt-out chart value, happy to add one.

TODOs:

  • squashed commits
  • includes documentation
  • adds unit tests

🤖 Generated with Claude Code

@kubernetes-prow kubernetes-prow Bot added kind/feature Categorizes issue or PR as related to a new feature. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jul 15, 2026
@kubernetes-prow

Copy link
Copy Markdown

This issue is currently awaiting triage.

If secrets-store-csi-driver contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubernetes-prow

Copy link
Copy Markdown

Welcome @aaresh-sharma!

It looks like this is your first PR to kubernetes-sigs/secrets-store-csi-driver 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-sigs/secrets-store-csi-driver has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

@kubernetes-prow kubernetes-prow Bot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Jul 15, 2026
@kubernetes-prow

Copy link
Copy Markdown

Hi @aaresh-sharma. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@linux-foundation-easycla

linux-foundation-easycla Bot commented Jul 15, 2026

Copy link
Copy Markdown

CLA Signed
The committers listed above are authorized under a signed CLA.

  • ✅ login: aaresh-sharma / name: Aaresh Sharma (2360eac)

@kubernetes-prow

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: aaresh-sharma
Once this PR has been reviewed and has the lgtm label, please assign aramase for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added the cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. label Jul 15, 2026
@kubernetes-prow
kubernetes-prow Bot requested review from aramase and enj July 15, 2026 17:00
@kubernetes-prow kubernetes-prow Bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 15, 2026
Pods that mount SecretProviderClass volumes can be scheduled onto a
node before the secrets-store CSI driver pod is running there, and
fail to mount until it is. Cluster operators can now close this race
by tainting nodes with secrets-store.csi.k8s.io/agent-not-ready at
registration (for example via EKS Managed Node Group taints). The
driver removes that taint from its own node once it starts, so
workload pods only schedule after the driver is ready.

The mechanism follows the aws-fsx-csi-driver implementation: a
background goroutine reads the node name from KUBE_NODE_NAME, gets
the Node, and patches out any taint with the matching key, retrying
with exponential backoff. The JSON patch pairs a test op on
/spec/taints with the replace so a concurrent taint update rejects
the patch and the retry re-reads fresh state. On clusters that never
apply the taint, or when KUBE_NODE_NAME is unset, this is a no-op.

RBAC adds nodes get/patch to the ClusterRole (staging chart and
kustomize; released copies promoted at release time).
@aaresh-sharma

Copy link
Copy Markdown
Author

/easycla

@kubernetes-prow kubernetes-prow Bot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Jul 15, 2026
@aaresh-sharma

Copy link
Copy Markdown
Author

/assign @aramase

@enj enj moved this to Subprojects - Needs Triage in SIG Auth Jul 16, 2026
@enj enj added this to SIG Auth Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/feature Categorizes issue or PR as related to a new feature. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants