[BUG] v2 instance-manager pod stuck in create/delete loop when engine frontend recovery blocks gRPC startup

[derekbit]

Describe the Bug

When an instance-manager pod (v2 data engine) restarts while persisted engine frontend records reference NVMe-TCP targets that are no longer reachable (e.g., deleted volume, migrated engine), the recoverEngineFrontends() function blocks NewServer() synchronously. This prevents gRPC servers from starting, causing the liveness probe to fail and kubelet to kill the container. Since the pod has restartPolicy: Never, the Longhorn controller deletes and recreates the pod, but the same stale metadata still exists, creating an infinite crash loop.

To Reproduce

1. Use Longhorn v1.12.x with v2 (SPDK) data engine enabled
2. Create ten v2 volumes and attach them, then delete it while the instance-manager pod is being recycled (or force-kill the IM pod while the volume is being deleted)
3. The instance-manager pod enters a create/delete loop.

Expected Behavior

The instance-manager pod should start serving gRPC immediately regardless of engine frontend recovery status. Recovery of stale/unreachable targets should fail fast and clean up without affecting pod health.

Support Bundle for Troubleshooting

N/A

Environment

• Longhorn version:
• Impacted volume (PV):
• Installation method (e.g. Rancher Catalog App/Helm/Kubectl):
• Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version:
  • Number of control plane nodes in the cluster:
  • Number of worker nodes in the cluster:
• Node config
  • OS type and version:
  • Kernel version:
  • CPU per node:
  • Memory per node:
  • Disk type (e.g. SSD/NVMe/HDD):
  • Network bandwidth between the nodes (Gbps):
• Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal):
• Number of Longhorn volumes in the cluster:

Additional context

No response

Workaround and Mitigation

No response

[longhorn-io-github-bot]

Pre Ready-For-Testing Checklist

• Context & Reproduction
  Where are the reproduction or test steps documented?
  Link:

  1. Deploy Longhorn and enable v2 data engine
  2. Create an folder /var/lib/longhorn/metadata/enginefrontends
  3. Manually create multiple metadata files on one of nodes like /var/lib/longhorn/metadata/enginefrontends/pvc-b9dcf347-9ccd-4dfd-917a-68a414c5e0b1/pvc-b9dcf347-9ccd-4dfd-917a-68a414c5e0b1/enginefrontend.json with the content

     {
       "name": "pvc-b9dcf347-9ccd-4dfd-917a-68a414c5e0b1-ef-0",
       "engineName": "pvc-b9dcf347-9ccd-4dfd-917a-68a414c5e0b1-e-0",
       "volumeName": "pvc-b9dcf347-9ccd-4dfd-917a-68a414c5e0b1",
       "volumeNqn": "nqn.2023-01.io.longhorn.spdk:volume-pvc-b9dcf347-9ccd-4dfd-917a-68a414c5e0b1",
       "volumeNguid": "338b0d0469d852b7a4ad684421793e1c",
       "frontend": "spdk-tcp-blockdev",
       "specSize": 501219328,
       "targetIP": "10.42.2.122",
       "targetPort": 21138,
       "activePath": "10.42.2.122:21138",
       "preferredPath": "10.42.2.122:21138",
       "paths": [
         {
           "targetIP": "10.42.2.122",
           "targetPort": 21138,
           "engineName": "pvc-b9dcf347-9ccd-4dfd-917a-68a414c5e0b1-e-0",
           "nqn": "nqn.2023-01.io.longhorn.spdk:volume-pvc-b9dcf347-9ccd-4dfd-917a-68a414c5e0b1",
           "nguid": "338b0d0469d852b7a4ad684421793e1c",
           "anaState": "optimized"
         }
       ]
     }
     

  4. Delete the v2 IM pod of the nodes
  5. Before the fix, the IM pod will be in create-and-delete loop
  6. After the fix, the IM pod will be started and running without issues.
  7. Creating and attaching v2 volumes works as well.

• Workaround
  Is there a workaround for the issue? If yes, where is it documented?
  Link:

• Fix / Feature Explanation
  Does the PR include a clear explanation of the fix or the new feature?

• Code Changes
  Have all required code changes been merged?
  PR(s): fix(spdk): make engine frontend recovery async to prevent pod crash loop longhorn-spdk-engine#539

• Deployment Changes (YAML / Helm Chart)
  Does the PR introduce any deployment changes?
  PR:

• Longhorn Enhancement Proposal (LEP) (if labeled require/LEP)
  Has the LEP PR been submitted?
  PR:

• Documentation Updates (if labeled require/doc)
  Has the required documentation PR been submitted or merged?
  PR:

• Manual Test Plan (if labeled require/manual-test-plan)
  Has the manual test plan been documented?
  Link: