ontap-nas-economy: ControllerPublish reassigns the FlexVol export policy to the publishing backend's policy, so Trident instances sharing an SVM and storagePrefix revoke each other's NFS access on every publish

[notsrch]

Describe the bug

With autoExportPolicy: true, every ControllerPublishVolume on an ontap-nas-economy backend
does two things to export policies (publishQtreeShare in storage_drivers/ontap/ontap_nas_qtree.go):

1. adds the node's rule to the qtree's own policy and assigns that policy to the qtree, and
2. adds the node's rule to the backend-wide policy trident-<backendUUID> and assigns that
   policy to the FlexVol that hosts the qtree (ensureNodeAccessForPolicyAndApply(..., backendPolicyName, applyFlexvolPolicy)).

Step 2 assumes the FlexVol belongs to this backend alone. The driver picks FlexVol pools by name
(trident_qtree_pool_<storagePrefix>*) and reuses any pool in the SVM with headroom, without
checking who created it or whose qtrees it already holds. When two or more Trident instances use
the same SVM and the same storagePrefix, they share pools, and every publish by one instance
reassigns the shared FlexVol to its trident-<backendUUID> policy, which lists only its node
IPs. ONTAP evaluates the FlexVol policy at mount time (junction traversal), so every qtree in that
FlexVol becomes unmountable for every other instance's nodes. The last publisher wins; the others
stay locked out until they publish into the same pool and lock the first one out in turn.

Nothing on the losing side looks wrong: its backend policy, its qtree policies,
tridentvolumepublications and tridentnodes are all correct. The only visible symptom is
NodePublishVolume failing with mount.nfs: access denied / exit status 32 for every new
mount of every volume in that pool, while established mounts keep working. Deleting the pod's
VolumeAttachment "fixes" it by forcing a publish, which flips the FlexVol back and breaks the
other instances.

I know the documentation says instances sharing an SVM must use distinct prefixes. This report
is about Trident failing unsafe when that rule is broken: it silently adopts pools it did not
create, then repeatedly revokes other tenants' access with no warning on either side. We found it
after a week of chasing what looked like #1179/#1181 lockouts.

Environment

• Trident version: 26.06.0 (the code path is unchanged on master)
• Trident installation flags used: operator-based install, defaults, enableConcurrency: true
• Container runtime: containerd (RKE2)
• Kubernetes version: v1.34.x
• Kubernetes orchestrator: RKE2
• Kubernetes enabled feature gates: defaults
• OS: Linux
• NetApp backend types: ONTAP 9.17.1P1, ontap-nas-economy, REST, autoExportPolicy: true,
  NFSv3; ~17 Kubernetes clusters, each with its own Trident, all with storagePrefix: pvc_nfs_economy_ on the same SVM

To Reproduce

Deterministic with two clusters:

1. Two Kubernetes clusters, each with its own Trident and an ontap-nas-economy backend on the
   same SVM with the same storagePrefix and autoExportPolicy: true.
2. Cluster A creates and publishes an RWX PVC. Note the FlexVol its qtree landed in and that the
   FlexVol's nas.export_policy is trident-<A's backendUUID>, containing A's node IPs.
3. Cluster B creates and publishes a PVC. If B's driver has room in the same pool it reuses it
   (it will, the pool just needs headroom), and the FlexVol's nas.export_policy becomes
   trident-<B's backendUUID>, containing only B's node IPs.
4. On cluster A, start a new pod for the PVC from step 2 on a node that already has a
   VolumeAttachment (so no new publish happens). MountVolume.SetUp failed ... error mounting NFS volume <lif>:/<flexvol>/<qtree> ... exit status 32, indefinitely. A's records are all correct.
5. Publish anything from A into that pool (e.g. delete a VolumeAttachment and let the pod
   recreate). A's mounts work again; now B's new mounts fail.

Observed on our arrays (names replaced):

FlexVol trident_qtree_pool_<prefix>_XXXXXXXXXX
  nas.export_policy = trident-<backendUUID of cluster P>     <- rules: one IP, cluster P's node
  qtrees from 17 different clusters' Tridents

cluster G's own state, same moment:
  trident-<backendUUID of cluster G>      6 rules, all of G's workers   (intact)
  qtree policy for G's shared volume      6 rules, all of G's workers   (intact)
  tridentvolumepublications / tridentnodes correct
  every new mount of G's volumes in that FlexVol: exit status 32

Proxy-side HTTP trace of the array, one pool:

21:13:03  cluster G  PATCH /api/storage/volumes/<flexvol-uuid>   (publish: policy -> trident-<G>)
21:52:47  cluster P  PATCH /api/storage/volumes/<flexvol-uuid>   (publish: policy -> trident-<P>)

Each PATCH is the 82-byte {"nas":{"export_policy":{"name":"trident-…"}}} body that
applyFlexvolPolicy sends.

Expected behavior

Any of the following would make this fail safe:

• ensureFlexvolForQtree should not reuse a pool it did not create. Trident could tag the
  FlexVols it creates (comment or label) and skip pools without its tag, or at least skip pools
  that already hold qtrees whose export policy is a trident-* policy of another backend, and log
  why.
• publishQtreeShare should not reassign a FlexVol that already carries a different trident-*
  policy. Adding the node's rule to the existing policy is not enough on its own, because the
  owning backend's periodic reconcile removes rules it does not consider desired; a shared FlexVol
  needs a FlexVol-scoped policy with add-only semantics, or the reconcile has to leave rules it
  did not add alone.
• Failing that, a loud warning at publish time when the FlexVol's current policy is a foreign
  trident-* policy, so the operator learns about the collision before an outage.

Additional context

• This compounds autoExportPolicy: a node with an existing publication is locked out permanently when its export rule is missing or stale — no code path ever repairs export policies #1181: the FlexVol-level policy has no repair path other than a fresh publish,
  and here the fresh publish is exactly what breaks the other tenant.
• It also confused the analysis of ontap-nas-economy: controller restart deletes all rules from the backend export policy, revoking NFS access for every node on the backend #1179 on our side: a Terminate during a backend update can
  succeed in deleting trident-<backendUUID> (no 1703945 rejection) precisely because the
  FlexVol no longer references it, having been reassigned by another instance.

AI assistance

Assisted-by: Claude Fable 5 noreply@anthropic.com