failed to pull cf-operator images from private docker registry

[naveenspen14]

Hi Team,

It is regarding pulling cf-operator images from private docker registry.

Currently we are using kubecf version 2.6.1 and its cf-operator.

We could able to modify kubecf values.yaml pointing to private docker registry through image_pull_secrets: [regcred]. It is successfully working.

But cf-operator is unable to pull the quark images from private docker registry.

k get pods -n cf-operator
NAME READY STATUS RESTARTS AGE
cf-operator-86d5d5d746-hzqpk 1/1 Running 0 7m25s
cf-operator-quarks-job-b86dd6bd8-gbmq2 0/1 ImagePullBackOff 0 7m25s
cf-operator-quarks-secret-66557b48c4-kw8rf 0/1 ImagePullBackOff 0 7m25s

It says authentication required.

We have provided the required credentials in values.yaml file ‘global’ section.

image:

repository that provides the operator docker image.

repository: cf-operator

org that provides the operator docker image.

org: myregistry/cfcontainerization

tag of the operator docker image

tag: v6.1.17-0.gec409fd7

global:

Context Timeout for each K8's API request in seconds.

contextTimeout: 300

MeltdownDuration is the duration (in seconds) of the meltdown period, in which we

postpone further reconciles for the same resource

meltdownDuration: 60

MeltdownRequeueAfter is the duration (in seconds) for which we delay the requeuing of the reconcile

meltdownRequeueAfter: 30
image:
# pullPolicy defines the policy used for pulling docker images.
pullPolicy: IfNotPresent
# credentials is used for pulling docker images.
credentials:
username: “xxxxxxxxxx”
password: “xxxxxxxxxx”
servername: "myregistry"

Updated registry in charts/quarks-job/values.yaml

image:
org: myregistry/cfcontainerization
repository: quarks-job
tag: v1.0.206

Update registry n charts/quarks-secret/values.yaml

image:
org: myregistry/cfcontainerization
repository: quarks-secret
tag: v1.0.744

**cf-operator-86d5d5d746-hzqpk ** pod describe

Able to pull the image from registry

Events:
Type Reason Age From Message

---

Normal Scheduled 37s default-scheduler Successfully assigned cf-operator/cf-operator-86d5d5d746-hzqpk to k8s-worker-1-kubecf288
Normal Pulling 36s kubelet, k8s-worker-1-kubecf288 Pulling image "myregistry/cfcontainerization/cf-operator:v6.1.17-0.gec409fd7"
Normal Pulled 22s kubelet, k8s-worker-1-kubecf288 Successfully pulled image "myregistry/cfcontainerization/cf-operator:v6.1.17-0.gec409fd7"
Normal Created 22s kubelet, k8s-worker-1-kubecf288 Created container cf-operator
Normal Started 22s kubelet, k8s-worker-1-kubecf288 Started container cf-operator

cf-operator-quarks-job-b86dd6bd8-gbmq2 pod describe
Image pull failed
Events:
Type Reason Age From Message

---

Normal Scheduled 7m6s default-scheduler Successfully assigned cf-operator/cf-operator-quarks-job-b86dd6bd8-gbmq2 to k8s-worker-1-kubecf288
Normal SandboxChanged 7m4s kubelet, k8s-worker-1-kubecf288 Pod sandbox changed, it will be killed and re-created.
Warning Failed 6m24s (x3 over 7m5s) kubelet, k8s-worker-1-kubecf288 Failed to pull image “myregistry/cfcontainerization/quarks-job:v1.0.206": rpc error: code = Unknown desc = Error response from daemon: Head https://myregistry/v2/cfcontainerization/quarks-job/manifests/v1.0.206: unknown: Authentication is required
Warning Failed 6m24s (x3 over 7m5s) kubelet, k8s-worker-1-kubecf288 Error: ErrImagePull
Warning Failed 5m48s (x7 over 7m3s) kubelet, k8s-worker-1-kubecf288 Error: ImagePullBackOff
Normal Pulling 5m37s (x4 over 7m5s) kubelet, k8s-worker-1-kubecf288 Pulling image “myregistry/cfcontainerization/quarks-job:v1.0.206"
Normal BackOff 2m2s (x23 over 7m3s) kubelet, k8s-worker-1-kubecf288 Back-off pulling image “myregistry/cfcontainerization/quarks-job:v1.0.206"

cf-operator-quarks-secret-66557b48c4-kw8rf pod describe

Image pull failed

Events:
Type Reason Age From Message

---

Normal Scheduled 10m default-scheduler Successfully assigned cf-operator/cf-operator-quarks-secret-66557b48c4-kw8rf to k8s-worker-2-kubecf288
Normal Pulling 8m40s (x4 over 10m) kubelet, k8s-worker-2-kubecf288 Pulling image “myregistrycfcontainerization/quarks-secret:v1.0.744"
Warning Failed 8m40s (x4 over 10m) kubelet, k8s-worker-2-kubecf288 Failed to pull image “myregistry/cfcontainerization/quarks-secret:v1.0.744": rpc error: code = Unknown desc = Error response from daemon: Head https://myregistry/v2/cfcontainerization/quarks-secret/manifests/v1.0.744: unknown: Authentication is required
Warning Failed 8m40s (x4 over 10m) kubelet, k8s-worker-2-kubecf288 Error: ErrImagePull
Warning Failed 8m15s (x7 over 10m) kubelet, k8s-worker-2-kubecf288 Error: ImagePullBackOff
Normal BackOff 8s (x42 over 10m) kubelet, k8s-worker-2-kubecf288 Back-off pulling image “myregistry/cfcontainerization/quarks-secret:v1.0.744"

Suggestions/ help required to pull images from private docker registry.

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/177015851

The labels on this github issue will be updated when the story is started.

[naveenspen14]

I fixed this issue, after adding the below changes to both images (cf-operator-quarks-job and cf-operator-quarks-secret) it worked like charm.

1. Included global.image.credentials in values.yaml file for both the images (quarks-job and quarks-secret)
2. added/created service-account-pull-secret.yaml for both under the templates directory
3. calling image credentials from secret (created in 2nd step) in service-account.yaml

My suggession is why can't we have a common secret file for pulling all images in cf-operator.

[manno]

My suggession is why can't we have a common secret file for pulling all images in cf-operator.

Maybe you could make a global.image.credentials.secretName option and change the qjob,qsec,qsts templates to use that name?

I'm not sure why the current pull secret name includes the service account name. In case one does install multiple operators in the same cluster?

I don't think quarks operator should support multiple operators in one namespace, so it's fine to have static name for the image pull secret.