Graceful shutdown fsnotify watcher

[Jefftree]

What type of PR is this?

/kind bug

What this PR does / why we need it:

The fsnotify watcher has a Run() function that fetches indefinitely with no way to stop, resulting in a goroutine leak:

https://github.com/kubernetes/kubernetes/blob/master/pkg/util/filesystem/watcher.go#L77-L93

Which issue(s) this PR is related to:

Fixes #135076

Special notes for your reviewer:

Does this PR introduce a user-facing change?

NONE

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

/sig storage

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Jefftree
Once this PR has been reviewed and has the lgtm label, please assign wojtek-t for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[Jefftree]

/assign @tchap @jpbetz

Attempted a fix for the leak which now passes the leak check test in #134990. I'm not very familiar with this code path though, wondering if someone from storage could take a look? cc @carlory?

[carlory]

I prefer to add a ctx parameter to the existing Run() function in the FSWatcher interface. It seems many files need to be changed to pass the ctx parameter to the Run() function. But it can avoid forgetting to call the Close() function. i.e., https://github.com/kubernetes/kubernetes/blob/master/cmd/kube-proxy/app/options.go#L290-L391