fix: allow multiple CIFS krb5 mounts with the same CRUID by kaidohTips · Pull Request #1052 · kubernetes-csi/csi-driver-smb

kaidohTips · 2026-04-02T20:50:38Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

This PR fixes a critical bug in the ensureKerberosCache function that occurs when a single pod attempt to mount multiple -kerberos volumes simultaneously using the same cruid

Problem:

When kubelet triggers parallel mounts for multiple volumes sharing the same cruid, multiple threads enter ensureKerberosCache simultaneously:

The original code attempts to os.Remove() the krb5cc_<cruid> symlink if it exists.
This means one thread can destroy a valid symlink just before another thread executes the Linux mount command, resulting in mount error(126): Required key not available.
Multiple threads attempt to call os.Symlink() at the exact same millisecond, causing subsequent threads to crash with a file exists error, which aborts the mount process.

Solution implemented:

Added an os.Stat check to verify if the symlink already exists and points to a valid file. If it does, the thread leaves it intact instead of removing it, preventing threads from sabotaging each other.
Added handling of the os.IsExist(err) error during os.Symlink(). If the error is raised, it simply means a concurrent thread just created the symlink a microsecond earlier, allowing the mount process to continue successfully.

Which issue(s) this PR fixes:
Fixes # None

Requirements:

uses conventional commit messages
- includes documentation
adds unit tests
tested upgrade from previous version

Special notes for your reviewer:
This fix was extensively tested on a production cluster
I scheduled a pod in a(JupyterHub environment) configured with 4 simultaneous volumes PersistentVolumes using sec=krb5 and the same cruid.

Before this patch: The parallel mounts consistently failed with file exists or Required key not available errors
After this patch: All 4 volumes mount instantly and reliably without any collision

Release note:

Fix kerberos cache symlink creation causing multiples SMB mounts to fail

k8s-ci-robot · 2026-04-02T20:50:47Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: kaidohTips
Once this PR has been reviewed and has the lgtm label, please assign msau42 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

linux-foundation-easycla · 2026-04-02T20:50:48Z

The committers listed above are authorized under a signed CLA.

✅ login: andyzhangx / name: andyzhangx (1a9117b, 1ce8689, 37af76d, 4142d2d, 95e443c, fe04017)
✅ login: kaidohTips / name: KaidohTips (1816f36, 4f7030b, 757de63, a15ec1f, dc25917, f3cf7c6)
✅ login: kaidohTips / name: kaidohTips (84eace0, 8a77fbd)

k8s-ci-robot · 2026-04-02T20:50:50Z

Welcome @kaidohTips!

It looks like this is your first PR to kubernetes-csi/csi-driver-smb 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes-csi/csi-driver-smb has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2026-04-02T20:50:52Z

Hi @kaidohTips. Thanks for your PR.

I'm waiting for a kubernetes-csi member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work.

Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Copilot

Pull request overview

This PR updates the SMB CSI node server’s Kerberos cache/symlink setup to avoid race conditions when kubelet mounts multiple sec=krb5 volumes in parallel using the same cruid.

Changes:

Avoids deleting an existing krb5cc_<cruid> path when it already exists.
Treats os.Symlink() “already exists” errors as non-fatal to tolerate concurrent creators.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

andyzhangx · 2026-04-03T05:03:53Z

@kaidohTips can you sign the easycla first? (select individual contributor)

kaidohTips · 2026-04-03T08:05:10Z

@kaidohTips can you sign the easycla first? (select individual contributor)

Hello @andyzhangx,

Done.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

os.Stat follows the symlink and reports ENOENT when the target is gone, causing ensureKerberosCache to skip the Remove path and then fail with "file exists" in os.Symlink. Switch to os.Lstat so the symlink itself is inspected regardless of target validity. Add TestEnsureKerberosCacheConcurrent: runs parallel goroutines against ensureKerberosCache with a shared CRUID and asserts the final symlink points at a valid volume-specific cache file, locking in the per-CRUID serialization contract.

ensureKerberosCache relies on Linux-specific CIFS/Kerberos semantics (os.Chown with cruid as gid, symlink handling). Restrict the test to Linux only to avoid failures on macOS and Windows CI runners.

andyzhangx

/ok-to-test

- Use os.Lstat + os.Readlink to properly validate symlinks instead of os.Stat - Accept any existing symlink pointing to a valid cache file (concurrent mount safe) - Use atomic temp-symlink + os.Rename instead of remove-then-create (no gap) - Remove TryAcquire lock that caused concurrent mounts to fail with Aborted - Include rich context (link name, target, credUID) in all error messages - Ensure consistent tab indentation (gofmt) - Update tests: valid symlink from previous volume is now kept, concurrent test no longer expects codes.Aborted

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Update function comment to accurately describe symlink reuse behavior - Only retry rename on os.IsExist; fail fast on permission/IO errors - Update test comment to reflect filesystem-based concurrency (no CRUID lock)

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Remove pre-removal of krb5CacheFileName; rely on os.Rename atomic replace - Skip ensureKerberosCache tests when non-root and uid != gid (Chown EPERM)

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…ache file

Copilot

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

If krb5CacheFileName is unexpectedly a directory, os.Rename cannot atomically replace it with a symlink. Explicitly remove directories before attempting the temp-symlink + rename pattern.

k8s-ci-robot · 2026-04-20T13:28:35Z

@kaidohTips: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-csi-driver-smb-e2e-windows-2022-hostprocess	`4142d2d`	link	true	`/test pull-csi-driver-smb-e2e-windows-2022-hostprocess`
pull-csi-driver-smb-e2e-windows-2022	`4142d2d`	link	false	`/test pull-csi-driver-smb-e2e-windows-2022`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Allow multiple CIFS krb5 mounts with the same CRUID

dc25917

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Apr 2, 2026

k8s-ci-robot requested review from andyzhangx and jingxu97 April 2, 2026 20:50

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 2, 2026

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Apr 2, 2026

andyzhangx requested a review from Copilot April 3, 2026 03:29

Copilot started reviewing on behalf of andyzhangx April 3, 2026 03:30 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go Outdated

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. labels Apr 3, 2026

andyzhangx requested a review from Copilot April 3, 2026 09:28

Copilot started reviewing on behalf of andyzhangx April 3, 2026 09:29 View session

Copilot AI reviewed Apr 3, 2026

View reviewed changes

Comment thread pkg/smb/nodeserver.go Outdated

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Apr 3, 2026

kaidohTips force-pushed the master branch from 3ab2254 to 9be9fcb Compare April 3, 2026 23:40

Fix commentary typo

84eace0

Copilot AI reviewed Apr 11, 2026

View reviewed changes

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go

andyzhangx changed the title ~~Allow multiple CIFS krb5 mounts with the same CRUID~~ fix: allow multiple CIFS krb5 mounts with the same CRUID Apr 11, 2026

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Apr 15, 2026

fix: skip TestEnsureKerberosCacheConcurrent on non-Linux platforms

1816f36

ensureKerberosCache relies on Linux-specific CIFS/Kerberos semantics (os.Chown with cruid as gid, symlink handling). Restrict the test to Linux only to avoid failures on macOS and Windows CI runners.

andyzhangx reviewed Apr 18, 2026

View reviewed changes

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Apr 18, 2026

fix: fix CRUID lock error and add ensureKerberosCache coverage

f3cf7c6

andyzhangx removed the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Apr 20, 2026

andyzhangx requested a review from Copilot April 20, 2026 08:51

Copilot started reviewing on behalf of andyzhangx April 20, 2026 08:52 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver.go

Comment thread pkg/smb/nodeserver_test.go Outdated

Address review comments (round 2)

fe04017

- Update function comment to accurately describe symlink reuse behavior - Only retry rename on os.IsExist; fail fast on permission/IO errors - Update test comment to reflect filesystem-based concurrency (no CRUID lock)

andyzhangx requested a review from Copilot April 20, 2026 09:09

Copilot started reviewing on behalf of andyzhangx April 20, 2026 09:09 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

Comment thread pkg/smb/nodeserver.go Outdated

Comment thread pkg/smb/nodeserver_test.go

Address review comments (round 3)

95e443c

- Remove pre-removal of krb5CacheFileName; rely on os.Rename atomic replace - Skip ensureKerberosCache tests when non-root and uid != gid (Chown EPERM)

andyzhangx requested a review from Copilot April 20, 2026 10:10

Copilot started reviewing on behalf of andyzhangx April 20, 2026 10:11 View session

Copilot AI reviewed Apr 20, 2026

View reviewed changes

Comment thread pkg/smb/nodeserver_test.go

andyzhangx added 2 commits April 20, 2026 10:19

Tighten concurrent test: verify symlink target is a volume-specific c…

37af76d

…ache file

Fix test: symlink target uses base64-encoded volumeID, not plain text

1ce8689

andyzhangx requested a review from Copilot April 20, 2026 10:45

Copilot AI reviewed Apr 20, 2026

View reviewed changes

Comment thread pkg/smb/nodeserver.go Outdated

Handle directory at kerberos cache path before atomic rename

4142d2d

If krb5CacheFileName is unexpectedly a directory, os.Rename cannot atomically replace it with a symlink. Explicitly remove directories before attempting the temp-symlink + rename pattern.

k8s-ci-robot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Apr 20, 2026

Conversation

kaidohTips commented Apr 2, 2026 • edited by andyzhangx Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 2, 2026

Uh oh!

linux-foundation-easycla Bot commented Apr 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 2, 2026

Uh oh!

k8s-ci-robot commented Apr 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andyzhangx commented Apr 3, 2026

Uh oh!

kaidohTips commented Apr 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

andyzhangx left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

k8s-ci-robot commented Apr 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kaidohTips commented Apr 2, 2026 •

edited by andyzhangx

Loading

linux-foundation-easycla Bot commented Apr 2, 2026 •

edited

Loading