Fix for VSO Stops Reconciling for VSS After a Transient Vault HA Event

[tejashwiniingalagi]

Description:

Problem
After a Vault HA event (leader re-election, pod restart), VSO emits transient errors like:

"local node not active but active cluster node not found"

This is expected and self-resolving — once the new leader is established, Vault recovers. However, VSO permanently stopped reconciling the affected VaultStaticSecret resources. Days later, secrets were stale with zero controller log entries since the failure. Restarting the VSO pod alone did not recover them; the only workaround was a manual spec-touch to force a re-enqueue.

Root Cause
Two error return paths in VaultStaticSecretReconciler.Reconcile returned nil error alongside RequeueAfter:

// Before — both error paths
return ctrl.Result{RequeueAfter: horizon}, nil

controller-runtime treats a nil error + RequeueAfter as a one-shot delayed retry. Once that timer fires and Vault has not yet recovered, no further retries are scheduled — the resource is silently dropped from the work queue.

A secondary issue: when hmacSecretData=true and the Vault read succeeds but the secret data is unchanged (doSync=false), the SecretSynced condition was never updated — leaving it permanently False even after full recovery.

Fix

1. Return the non-nil error on both Vault error paths (ClientFactory.Get and c.Read):

// After
return ctrl.Result{RequeueAfter: horizon}, err

Returning a non-nil error alongside RequeueAfter tells controller-runtime to place the resource back into the rate-limiter queue. It will keep retrying with exponential backoff via BackOffRegistry (5s → 60s plateau) indefinitely until Vault recovers — no spec-touch required.

2. Write SecretSynced=True in the doSync=false branch:

When HMAC comparison shows the secret data is unchanged, the reconciler previously wrote no condition at all — leaving a stale SecretSynced=False from the prior failure. Now both the sync and no-sync paths always write the condition, accurately reflecting the healthy post-recovery state.

PCI review checklist

• I have documented a clear reason for, and description of, the change I am making.

• If applicable, I've documented a plan to revert these changes if they require more than reverting the pull request.

• If applicable, I've documented the impact of any changes to security controls.

  Examples of changes to security controls include using new access control methods, adding or removing logging pipelines, etc.