driver is giving warnung error as "Unable to retrieve some image pull secrets" on k8s cluster

[saurabhwani5]

Describe the bug

In driver pod description of 2.11.0 then error is coming as "Unable to retrieve some image pull secrets"

How to Reproduce?

Please list the steps to help development teams reproduce the behavior

1. Install CSI 2.11.0 GM build :

[root@csi-kube129-1-x-master saurabh]# oc apply -f ibm-spectrum-scale-csi-operator.yaml
deployment.apps/ibm-spectrum-scale-csi-operator created
clusterrole.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator created
clusterrolebinding.rbac.authorization.k8s.io/ibm-spectrum-scale-csi-operator created
serviceaccount/ibm-spectrum-scale-csi-operator created
customresourcedefinition.apiextensions.k8s.io/csiscaleoperators.csi.ibm.com created
[root@csi-kube129-1-x-master saurabh]# oc apply -f csiscaleoperators.csi.ibm.com_cr.yaml
csiscaleoperator.csi.ibm.com/ibm-spectrum-scale-csi created
[root@csi-kube129-1-x-master saurabh]# oc get pods
NAME                                                  READY   STATUS    RESTARTS   AGE
ibm-spectrum-scale-csi-9wrmw                          3/3     Running   0          9s
ibm-spectrum-scale-csi-attacher-797bc6ff85-plgws      1/1     Running   0          9s
ibm-spectrum-scale-csi-attacher-797bc6ff85-rmg75      1/1     Running   0          9s
ibm-spectrum-scale-csi-d6zxv                          3/3     Running   0          9s
ibm-spectrum-scale-csi-operator-64bcd4dbd7-zvvcd      1/1     Running   0          52s
ibm-spectrum-scale-csi-provisioner-6dd44f9b4-qjm4j    1/1     Running   0          9s
ibm-spectrum-scale-csi-resizer-5999c8796f-4ms49       1/1     Running   0          9s
ibm-spectrum-scale-csi-snapshotter-5c5d7fbf5b-47zmx   1/1     Running   0          9s
[root@csi-kube129-1-x-master saurabh]# oc get cso
NAME                     VERSION   SUCCESS
ibm-spectrum-scale-csi   2.11.0    True
[root@csi-kube129-1-x-master saurabh]# oc describe pod | grep quay
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:b2bc343eadbc11d9ed74a8477d2cd0a7a8460a72203d3f6236d4662e68df1166
  Normal   Pulled                           18s                kubelet            Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:b2bc343eadbc11d9ed74a8477d2cd0a7a8460a72203d3f6236d4662e68df1166" already present on machine
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:b2bc343eadbc11d9ed74a8477d2cd0a7a8460a72203d3f6236d4662e68df1166
  Normal   Pulled                           18s                kubelet            Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:b2bc343eadbc11d9ed74a8477d2cd0a7a8460a72203d3f6236d4662e68df1166" already present on machine
    Image:         quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:bd264199ac10d574163bfa32bb88844fd786ee6f794a56e235591d2f051c7807
      CSI_DRIVER_IMAGE:      quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:b2bc343eadbc11d9ed74a8477d2cd0a7a8460a72203d3f6236d4662e68df1166
  Normal   Pulled                           61s                kubelet            Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-operator@sha256:bd264199ac10d574163bfa32bb88844fd786ee6f794a56e235591d2f051c7807" already present on machine

2. Check the driver pod description :

Events:
  Type     Reason                           Age                 From               Message
  ----     ------                           ----                ----               -------
  Normal   Scheduled                        4m2s                default-scheduler  Successfully assigned ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-d6zxv to csi-kube129-1-x-worker2.fyre.ibm.com
  Warning  FailedMount                      4m2s                kubelet            MountVolume.SetUp failed for volume "spectrum-scale-config" : configmap "spectrum-scale-config" not found
  Normal   Pulled                           4m1s                kubelet            Container image "quay.io/ibm-spectrum-scale-dev/ibm-spectrum-scale-csi-driver@sha256:b2bc343eadbc11d9ed74a8477d2cd0a7a8460a72203d3f6236d4662e68df1166" already present on machine
  Normal   Created                          4m1s                kubelet            Created container ibm-spectrum-scale-csi
  Normal   Started                          4m1s                kubelet            Started container ibm-spectrum-scale-csi
  Normal   Pulled                           4m1s                kubelet            Container image "registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:c53535af8a7f7e3164609838c4b191b42b2d81238d75c1b2a2b582ada62a9780" already present on machine
  Normal   Created                          4m                  kubelet            Created container driver-registrar
  Normal   Started                          4m                  kubelet            Started container driver-registrar
  Normal   Pulled                           4m                  kubelet            Container image "registry.k8s.io/sig-storage/livenessprobe@sha256:5baeb4a6d7d517434292758928bb33efc6397368cbb48c8a4cf29496abf4e987" already present on machine
  Normal   Created                          4m                  kubelet            Created container liveness-probe
  Normal   Started                          4m                  kubelet            Started container liveness-probe
  Warning  FailedToRetrieveImagePullSecret  30s (x6 over 4m1s)  kubelet            Unable to retrieve some image pull secrets (ibm-spectrum-scale-csi-registrykey, ibm-entitlement-key); attempting to pull the image may not succeed.

Expected behavior

on k8s env above error log should not come

Logs :

scale-csi/D.1138
csisnap.tar.gz

[keepalived03]

Unable to retrieve some image pull secrets (ibm-spectrum-scale-csi-registrykey, ibm-entitlement-key); attempting to pull the image may not succeed. 这种问题该怎么解决

[hemalathagajendran]

@keepalived03 In which CSI version do you see this error?

[zhujunzj]

@keepalived03 In which CSI version do you see this error?

K8S 1.28，Scale 5.1.9.6，CSI 2.13 has the same issue.

Controlled By: ReplicaSet/ibm-spectrum-scale-csi-operator-6dc9d786b6
Containers:
operator:
Container ID: containerd://41060ffd7f3ff1626d8723666fb3ac6d5c5f3aca08ce1805d5a0ad3927088256
Image: 10.200.53.208/tools/ibm-spectrum-scale-csi-operator:v2.13
Image ID: 10.200.53.208/tools/ibm-spectrum-scale-csi-operator@sha256:a1d9bde260fe500b6790b60a906a79b34d7b3db67931ff482acb61052d00fe09
Port:
Host Port:
Args:
--leaderElection=true
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: StartError
Message: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "--leaderElection=true": executable file not found in $PATH: unknown
Exit Code: 128

Tolerations: node-role.kubernetes.io/control-plane:NoSchedule op=Exists
node-role.kubernetes.io/infra:NoSchedule op=Exists
node-role.kubernetes.io/master:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message

---

Normal Scheduled 3m47s default-scheduler Successfully assigned ibm-spectrum-scale-csi-driver/ibm-spectrum-scale-csi-operator-6dc9d786b6-zp5vm to w64-worker-146.5
Normal Pulling 3m47s kubelet Pulling image "10.200.53.208/tools/ibm-spectrum-scale-csi-operator:v2.13"
Normal Pulled 106s kubelet Successfully pulled image "10.200.53.208/tools/ibm-spectrum-scale-csi-operator:v2.13" in 2m1.181s (2m1.181s including waiting)
Normal Created 88s (x3 over 106s) kubelet Created container operator
Warning Failed 88s (x3 over 106s) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "--leaderElection=true": executable file not found in $PATH: unknown
Normal Pulled 88s (x2 over 105s) kubelet Container image "10.200.53.208/tools/ibm-spectrum-scale-csi-operator:v2.13" already present on machine
Warning FailedToRetrieveImagePullSecret 77s (x9 over 3m47s) kubelet Unable to retrieve some image pull secrets (ibm-entitlement-key); attempting to pull the image may not succeed.
Warning BackOff 77s (x6 over 104s) kubelet Back-off restarting failed container operator in pod ibm-spectrum-scale-csi-operator-6dc9d786b6-zp5vm_ibm-spectrum-scale-csi-driver(55c0529b-e23b-4178-8bd5-4dfd1f954551)