Fall back to the mount table when NodeUnpublishVolume cannot stat the target path

[notsrch]

Change description

NodeUnpublishVolume (core/node/unmount.go, unmountGeneric) stats the target path before
it unmounts, to decide between the directory heuristic and the mount-table lookup. Only
not-found was tolerated; any other stat error was returned as codes.Internal before
umount was attempted.

On an NFS mount whose export rule no longer admits the node, stat on the mount point itself
returns EACCES. kubelet then retries the unpublish forever with the same result: the pod never
leaves Terminating, the node keeps the volume in status.volumesInUse, and the VolumeAttachment
is never detached. Force-deleting the pod does not help, and the attach/detach controller did not
force-detach within 10 minutes on Kubernetes v1.35. Issue #1184 has the details.

Fix: when stat fails with anything other than not-found, log a warning with the error and take
the existing mount-table branch (IsMounted reads /proc/self/mountinfo and never touches the
filesystem behind the path). Everything after that point is unchanged: umount, best-effort
removal of the target path, and tracking-file cleanup. Not-found still returns success as before.

The fallback covers every stat error rather than EACCES alone because ESTALE and EIO on a
stale share hit the same pre-check and are equally irrelevant to umount(2); the warning records
which one it was.

Why it matters beyond tidiness: with autoExportPolicy, a per-volume export rule that goes
missing is only written back by a fresh ControllerPublishVolume (#1181), which Kubernetes only
issues after the VolumeAttachment is gone. This bug pinned the VolumeAttachment for exactly the
node that needed the republish, so the missing rule denied I/O, the denied I/O failed the
unpublish, the failed unpublish held the attachment, and the held attachment blocked the repair.
The only exit was editing the export policy on the array.

Fixes #1184.

Project tracking

• This PR does not require a JIRA ticket (community contribution; details in the issue)

Testing

Summary: three new unit tests cover the fallback and fail against master; the bug was reproduced
deterministically on a live ONTAP cluster, on two 26.06.0-based builds, and the failing call is
the pre-check alone (the same unpublish succeeded on its next retry once access was restored).
With this change built into the same image the same step completes: one warning, unmount, pod
gone in 30 s, attachment released in 33 s.

Unit tests

• go build ./core/node/, go vet ./core/node/, gofmt -l core/node clean.
• go test ./core/node/ -count=1 green on the branch.
• New tests in core/node/unmount_test.go, all three fail against master's unmount.go:
  • TestUnmountGeneric_StatError_Mounted_FallsBackToMountTableAndUnmounts: stat returns
    EACCES, the mount table says mounted, umount is called, cleanup runs.
  • TestUnmountGeneric_StatError_NotMounted_SkipsUmount: stat fails, the mount table says
    not mounted, no umount, cleanup runs.
  • TestUnmountGeneric_StatError_MountTableError_Wrapped: stat fails and the mount-table
    lookup also fails; the error is returned wrapped, as the existing non-directory branch does.
• Existing not-found behaviour is unchanged and still covered.

Live reproduction (fails on 26.06.0-based builds)

Environment: 3-node RKE2 cluster, Kubernetes v1.35.4, Trident installed by the operator with
enableConcurrency: true. One ontap-nas-economy backend on ONTAP 9.17.1P1 (REST),
autoExportPolicy: true, NFSv3. One RWX PVC mounted by a DaemonSet, so every node holds a
publication and a VolumeAttachment. ONTAP queried and edited over REST from inside the cluster.

The node plugin under test was not stock 26.06.0. It was v26.06.0 plus the fixes for #1179
(from PR #1183) and #1181 (registration-driven export rule repair). Neither of those changes
touches core/node/ or the CSI node server, so the unpublish path was stock in both runs. The
combined build matters for the recovery observations below, not for the failure.

Trigger, from the issue: delete node-b's rule from the volume's qtree export policy on the array
(DELETE /api/protocols/nfs/export-policies/{id}/rules/{index}, HTTP 200).

Run 1 (v26.06.0 + #1179 fix only):

Time	Step	Observation
T+0	Rule deleted.	Established pod on node-b: read and write fail with Permission denied.
T+11 min	Deleted the DaemonSet pod on node-b.	Pod stuck in Terminating. Node plugin, on every kubelet retry: GRPC error: rpc error: code = Internal desc = could not check if the target path (/var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv>/mount) is a directory; stat ...: permission denied. Node status.volumesInUse still lists the volume; VolumeAttachment still attached.
T+14 min	kubectl delete pod --force --grace-period=0.	Pod object gone. kubelet keeps retrying the same failing unpublish; volumesInUse and the VolumeAttachment unchanged.
T+24 min	10 minutes after the force delete.	No forced detach by the attach/detach controller.
T+25 min	Rule re-added on the array (POST .../rules, HTTP 201).	Unpublish succeeded on its next retry; VolumeAttachment deleted 31 s later.

Run 2 (v26.06.0 + #1179 + #1181 fixes):

Time	Step	Observation
T+0	Rule deleted. 3 s later, deleted the DaemonSet pod on node-b.	Pod stuck in Terminating; 40 occurrences of the same stat ...: permission denied unpublish failure in the first 35 s; VolumeAttachment held. Identical to run 1.
T+1 min	Restarted the Trident node pod on node-b.	The #1181 fix repaired the qtree rule within seconds of the node registering. The pending unpublish then succeeded on its next retry, the stuck pod cleared, and the replacement DaemonSet pod came up Running.

What the two runs show:

• The failure is the stat pre-check alone. Nothing else changed between "fails on every retry"
  and "succeeds on the next retry" except that stat stopped returning EACCES.
• The autoExportPolicy: a node with an existing publication is locked out permanently when its export rule is missing or stale — no code path ever repairs export policies #1181 repair gives the deadlock an exit, but only when a node registers. That fix has no
  periodic loop, so a pod whose export access was revoked stays stuck until someone restarts the
  node's Trident pod or the node reboots. The unpublish should not depend on that; it should
  unmount a path it cannot read.

Retest with this change (passes)

Same cluster, backend, volume and DaemonSet as the reproduction. Image: v26.06.0 plus the #1179
fix, the #1181 fix, and this change. Clean state confirmed first: 3 rules in both policies, 3
publications, 3 VolumeAttachments, all pods Running.

Time	Step	Result
T+0	Deleted node-b's rule from the qtree policy on the array.	2 rules; established pod on node-b: write fails with Permission denied, as before.
T+3 s	Excluded node-b from the DaemonSet (so no replacement pod would hold the attachment), then deleted the DaemonSet pod on node-b.	Pod enters Terminating.
T+34 s	Node plugin on node-b, once: level=warning msg="Could not stat target path; checking the mount table instead." error="stat /var/lib/kubelet/pods/<uid>/volumes/kubernetes.io~csi/<pv>/mount: permission denied".	Unmount proceeds. Zero could not check if the target path errors.
T+36 s	Pod gone; VolumeAttachment for node-b deleted.	ControllerUnpublish ran: publications 2.
T+69 s	Node-b re-admitted.	New pod Running 10 s later; the fresh ControllerPublish rewrote the rule (3 rules); pod reads and writes.

No node registration happened during this run, so the #1181 repair path did not take part; the
recovery is this change alone. The not-found path was not separately exercised here; it is
covered by the existing unit test and is unchanged.

Comparison, same step on three builds:

Build	Delete the pod after revoking its node's qtree rule
v26.06.0 + #1179	Stuck in Terminating; unpublish fails every retry; force delete no help; no forced detach in 10 min; exit only by re-adding the rule on the array.
v26.06.0 + #1179 + #1181	Same loop (40 failures in 35 s); cleared only by restarting the node's Trident pod, which triggers the #1181 registration repair.
v26.06.0 + #1179 + #1181 + this change	One warning, unmount succeeds, pod gone in 30 s, attachment released in 33 s, next publish restores the rule.

Checklist

• Unit tests added
• CHANGELOG updated
• Documentation: none needed (no user-visible option changes)

AI assistance

Assisted-by: Claude Fable 5 noreply@anthropic.com