CSO events for gui user password expiry are not periodic

[saurabhwani5]

Describe the bug

CSO event when GUI User expired of remote cluster is coming before 24 Hours , where it should be after completion of 24 Hours.

How to Reproduce?

1. Install CSI 2.10 (PR1002 Image)

[root@saurabh6-master ~]# oc get pods
NAME                                                  READY   STATUS    RESTARTS        AGE
ibm-spectrum-scale-csi-attacher-b6b6d4948-5kfcn       1/1     Running   0               6h56m
ibm-spectrum-scale-csi-attacher-b6b6d4948-pz77p       1/1     Running   0               6h56m
ibm-spectrum-scale-csi-operator-567f8455f5-l7ntt      1/1     Running   1 (6h57m ago)   2d1h
ibm-spectrum-scale-csi-provisioner-b456fbb49-tzlsx    1/1     Running   0               6h56m
ibm-spectrum-scale-csi-resizer-84d84bfdf6-x798x       1/1     Running   0               6h56m
ibm-spectrum-scale-csi-sj7wj                          3/3     Running   0               6h56m
ibm-spectrum-scale-csi-snapshotter-656d4bd64f-s8fv9   1/1     Running   0               6h56m
ibm-spectrum-scale-csi-vbkvw                          3/3     Running   0               6h56m
[root@saurabh6-master ~]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.10.0    True
[root@saurabh6-master ~]# oc describe pod | grep quay
    Image:         quay.io/badri_pathak/ibm-spectrum-scale-csi-operator:events_gui_pass_v9
    Image ID:      quay.io/badri_pathak/ibm-spectrum-scale-csi-operator@sha256:a31ef1636903b3f3f75406cc463d0045a5064c7db12d59b07a322e89259e2065
      CSI_DRIVER_IMAGE:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:ee9bd3e431cf0d3fb1e407a6e2ed51d6be957dd1445c2cd88f329cbb5b1ea494
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:ee9bd3e431cf0d3fb1e407a6e2ed51d6be957dd1445c2cd88f329cbb5b1ea494
    Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:31172dc13f5cc514cf2474cb440697c1da20d035f3fd12ec761f12b06cc2e0a7
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:ee9bd3e431cf0d3fb1e407a6e2ed51d6be957dd1445c2cd88f329cbb5b1ea494
    Image ID:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:31172dc13f5cc514cf2474cb440697c1da20d035f3fd12ec761f12b06cc2e0a7

2. Expire the password of storage cluster which is being used by CSI
   [root@csi-86-scalegui ~]# /usr/lpp/mmfs/gui/cli/chuser remoteadmin --expirePassword
3. Check CSO event after 24 Hours

Events:
  Type     Reason         Age                  From              Message
  ----     ------         ----                 ----              -------
  Normal   CSIConfigured  17m (x9 over 24h)    CSIScaleOperator  The CSI driver resources have been created/updated successfully
  Warning  AuthError      17m (x3 over 3h48m)  CSIScaleOperator  Either the username/password is incorrect or the password has been expired for the Scale GUI clusterIds: [227604367530725699]

Expected behavior

CSO events should come after 24 hours only , as per above cso event is coming before 3h48m , Also this event is coming 3 times which is also not expected

Data Collection and Debugging

/scale-csi/D.1005

[badri-pathak]

The current manager uses default period which is 10hrs to execute Reconcile loop of the controller everytime.
The gui-password expiry check is being called on the every reconcile irrespect of the setting any RequeueAfter of 24 hours i.e. reconcile after every 24hrs because by deafult reconcile happens in 10hrs.
If needed to perform manual reconcile before auto reconcile loop we need to set SyncPeriod while manager creation of each controller and accordance to RequeueAfter in the Reconcile function.

e.g.

syncPeriod := time.Duration(4 * time.Minute)
mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
		Scheme:             scheme,
		MetricsBindAddress: metricsAddr,
		...
		NewCache:                cache.MultiNamespacedCacheBuilder(namespaces),
		SyncPeriod:              &syncPeriod,
	})

Reference manager options : https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.14.6/pkg/manager#Options

[badri-pathak]

Normal CSIConfigured 17m (x9 over 24h) CSIScaleOperator The CSI driver resources have been created/updated successfully
Warning AuthError 17m (x3 over 3h48m) CSIScaleOperator Either the username/password is incorrect or the password has been expired for the Scale GUI clusterIds: [227604367530725699]

The password expiry is being checked on every reconcile. Here the last event of password expiry has happened 17m ago, this could be because the reconcile has happened because of other reasons. The password expiry checks is not the reason for this non-periodic time. There are various reasons for reconciles.