Bug: etcd PVC reconcile blocks recovery from a failed Helm install

[sangheee]

Summary

When a Milvus CR is created, the operator runs helm install for the etcd release.
If that install fails and the StatefulSet is not created (e.g. a transient apiservice/discovery error), the failed reconcile is requeued and the retry takes the update path, which for etcd runs reconcilePVCs.
reconcilePVCs then fails because the StatefulSet doesn't exist, so the release can never recover and the CR is stuck forever.
A manual helm uninstall is the only way out. Introduced in #363.

Proposed fix

reconcilePVCs runs on every etcd update even when the size is unchanged (oldSize == newSize in our case), so it does all this work when there is nothing to resize.

milvus-operator/pkg/controllers/dependencies.go

Lines 258 to 273 in 3ae2cbf

	func (l *LocalHelmReconciler) reconcilePVCs(ctx context.Context, namespace, releaseName, oldSize, newSize string, mc v1beta1.Milvus) error {
	logger := ctrl.LoggerFrom(ctx)
	logger.Info("Reconciling PVCs", "namespace", namespace, "release", releaseName, "oldSize", oldSize, "newSize", newSize)
	stsName := releaseName
	newQuantity, err := resource.ParseQuantity(newSize)
	if err != nil {
	return fmt.Errorf("failed to parse new size: %v", err)
	}
	// Create K8sUtil instance for saving/getting objects
	k8sUtil := NewK8sUtil(l.mgr.GetClient())
	// Generate save name for the StatefulSet
	saveName := fmt.Sprintf("%s-old-sts", releaseName)

Option A

Skip when the size hasn't changed, before touching the StatefulSet
A failed install doesn't change the configured size, so this makes reconcilePVCs a no-op and lets helm.Update retry the install.

		if oldSizeStr != newSizeStr {
			logger.Info("reconcile PVC", "old size:", oldSizeStr, "new size:", newSizeStr, "release", request.ReleaseName)
			if err := l.reconcilePVCs(ctx, request.Namespace, request.ReleaseName, oldSizeStr, newSizeStr, mc); err != nil {
				return err
			}
		}

Option B

if there's no live and no saved StatefulSet, return nil instead of erroring so Helm can create it.

milvus-operator/pkg/controllers/dependencies.go

Lines 282 to 286 in 3ae2cbf

	savedSts := &appsv1.StatefulSet{}
	key := client.ObjectKey{Name: saveName, Namespace: namespace}
	err = k8sUtil.GetSavedObject(ctx, key, savedSts)
	if err != nil {
	return fmt.Errorf("failed to get saved StatefulSet: %v", err)

    if err := k8sUtil.GetSavedObject(ctx, key, savedSts); err != nil {
        // no live and no saved StatefulSet => nothing to resize; let Helm create it
        // logging skip message
        return nil
    }

I'd really appreciate it if you could look into it. Thank you!