reclaimspacejob: requeue with delay when node client is not found

When a ReclaimSpaceJob targets a PVC whose application pod has been
deleted, the VolumeAttachment may still show the volume as attached
while the node's CSI addons sidecar connection is no longer available
in the connection pool. This causes nodeReclaimSpace to fail with
"node client not found".

Previously, this error was returned directly to controller-runtime,
which requeues with a fast rate limiter (~5ms exponential backoff).
All 6 retries would exhaust in ~315ms, far too quickly for the
VolumeAttachment to be updated, causing the job to fail immediately.

Requeue with a 30-second interval for this specific error, giving
sufficient time for the VolumeAttachment to be cleaned up. On
subsequent reconciles, getTargetDetails will no longer find the
stale VolumeAttachment and the job can proceed with controller-only
reclaim space.

Signed-off-by: Madhu Rajanna <madhupr007@gmail.com>

Author: Madhu-1

internal/controller/csiaddons/reclaimspacejob_controller.go

@@ -59,8 +59,17 @@ const (
 	// failed reason type.
 	// TODO: add more useful reason types.
 	reasonFailed = "failed"
+
+	// nodeClientRequeueInterval is the interval to requeue when the
+	// node client is not found in the connection pool, allowing time
+	// for the VolumeAttachment to be updated.
+	nodeClientRequeueInterval = 30 * time.Second
 )
 
+// errNodeClientNotFound is a sentinel error returned when the node
+// client for the given nodeID is not found in the connection pool.
+var errNodeClientNotFound = errors.New("node client not found")
+
 // ReclaimSpaceJobReconciler reconciles a ReclaimSpaceJob object.
 type ReclaimSpaceJobReconciler struct {
 	client.Client
@@ -159,6 +168,13 @@ func (r *ReclaimSpaceJobReconciler) Reconcile(ctx context.Context, req ctrl.Requ
 		return ctrl.Result{}, nil
 	}
 
+	// If the node client is not found, requeue after an interval
+	// to allow the VolumeAttachment to be updated instead of using
+	// the fast rate-limited requeue.
+	if errors.Is(err, errNodeClientNotFound) {
+		return ctrl.Result{RequeueAfter: nodeClientRequeueInterval}, nil
+	}
+
 	return ctrl.Result{}, err
 }
 
@@ -445,7 +461,7 @@ func (r *ReclaimSpaceJobReconciler) nodeReclaimSpace(
 		return nil, err
 	}
 	if nodeClient == nil {
-		return nil, fmt.Errorf("node Client not found for %q nodeID", target.nodeID)
+		return nil, fmt.Errorf("%w for %q nodeID", errNodeClientNotFound, target.nodeID)
 	}
 	*logger = logger.WithValues("nodeClient", clientName)