[BUG] podman copy to internal crc registry fails connection refused

[ayates83]

General information

• OS: OSX 12.4 (21F79)
• Hypervisor: kvm? whatever crc installs for osx
• Did you run crc setup before starting it (Yes/No)? Yes
• Running CRC on: Laptop

CRC version

CRC version: 2.4.1+6b954d40
OpenShift version: 4.10.14
Podman version: 4.0.2

CRC status

 crc status --log-level debug
DEBU CRC version: 2.4.1+6b954d40                  
DEBU OpenShift version: 4.10.14                   
DEBU Podman version: 4.0.2                        
DEBU Running 'crc status'                         
DEBU Checking file: /Users/userid/.crc/machines/crc/.crc-exist 
DEBU Checking file: /Users/userid/.crc/machines/crc/.crc-exist 
DEBU Running SSH command: df -B1 --output=size,used,target /sysroot | tail -1 
DEBU Using ssh private keys: [/Users/userid/.crc/machines/crc/id_ecdsa /Users/userid/.crc/cache/crc_vfkit_4.10.14_amd64/id_ecdsa_crc] 
DEBU SSH command results: err: <nil>, output: 32737570816 18435547136 /sysroot 
DEBU Unexpected operator status for etcd: RecentBackup 
CRC VM:          Running
OpenShift:       Running (v4.10.14)
Podman:          
Disk Usage:      18.44GB of 32.74GB (Inside the CRC VM)
Cache Usage:     37GB
Cache Directory: /Users/userid/.crc/cache

CRC config

- consent-telemetry                     : no

Host Operating System

ProductName:	macOS
ProductVersion:	12.4
BuildVersion:	21F79

Steps to reproduce

1. oc login -u kubeadmin -p $pw https://https://api.crc.testing:6443
2. REG=$(oc get route -n openshift-image-registry | awk '{print $2}' | grep -v HOST)
3. podman login -u kubeadmin -p $(oc whoami -t)
4. podman tag localhost/image ${REG}/project/image:latest
5. podman push ${REG}/project/image:latest

Expected

expecting the copy to succeed to the internal registry

Actual

1. podman push ${REG}/project/image:latest
   Error: error pushing image "default-route-openshift-image-registry.apps-crc.testing/project/image:latest": trying to reuse blob sha256:5d256cdcff666d660337d13f80caec5485cb8997642ce5e97cf4191cb5a06585 at destination: pinging container registry default-route-openshift-image-registry.apps-crc.testing: Get "https://default-route-openshift-image-registry.apps-crc.testing/v2/": dial tcp 127.0.0.1:443: connect: connection refused

Logs

See attached debug log from crc start --log-level debug

[crc-debug.txt](https://github.com/code-ready/crc/files/9043737/crc-debug.txt)

[ayates83]

crc-debug.txt

[praveenkumar]

@ayates83 As per debug logs I can see following message which means cluster was not even running because of cert error

evel=debug msg="Running SSH command: timeout 5s oc get nodes --context admin --cluster crc --kubeconfig /opt/kubeconfig"
level=debug msg="SSH command results: err: Process exited with status 1, output: "
level=debug msg="Unable to connect to the server: x509: certificate has expired or is not yet valid: current time 2022-07-01T19:53:29Z is after 2022-07-01T03:52:10Z\n"

Can you download the latest version of crc 2.5.1 and try again?

[ayates83]

CRC version: 2.5.1+5b02b826
OpenShift version: 4.10.18
Podman version: 4.1.0

updated to crc 2.5.1 ; here's the debug log output
2crc-debug.txt

login successful:

podman login -u kubeadmin -p $(oc whoami -t) $REG --tls-verify=false
Login Succeeded!

push fails:

podman push ${REG}/project/image:latest
Error: error pushing image "default-route-openshift-image-registry.apps-crc.testing/project/image:latest": trying to reuse blob sha256:5d256cdcff666d660337d13f80caec5485cb8997642ce5e97cf4191cb5a06585 at destination: pinging container registry default-route-openshift-image-registry.apps-crc.testing: Get "https://default-route-openshift-image-registry.apps-crc.testing/v2/": dial tcp 127.0.0.1:443: connect: connection refused

same error occurs when trying --tls-verify=false as well.

[praveenkumar]

@ayates83 Can you provide the output of following?

$ oc get co
$ oc projects | grep <project_where_you_want_to_push>  // Make sure that project exist
$ oc get pods -n openshift-image-registry
NAME                                               READY   STATUS      RESTARTS   AGE
cluster-image-registry-operator-55fb9dcd9b-5qpcp   1/1     Running     0          22h
image-pruner-27620640-cw79d                        0/1     Completed   0          3h12m
image-registry-5bd79db9d6-szsn5                    1/1     Running     0          20h
node-ca-dtwc5                                      1/1     Running     0          47h
$ oc logs <image_registry_pod> -n openshift-image-registry | tail

[ayates83]

2022.07.07 22:15:24 [userid@host ~/git/osecli/ocp_resources] $ oc get co
NAME                                       VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                             4.10.18   True        False         False      11h     
config-operator                            4.10.18   True        False         False      16d     
console                                    4.10.18   True        False         False      11h     
dns                                        4.10.18   True        False         False      11h     
etcd                                       4.10.18   True        False         False      16d     
image-registry                             4.10.18   True        False         False      11h     
ingress                                    4.10.18   True        False         False      16d     
kube-apiserver                             4.10.18   True        False         False      16d     
kube-controller-manager                    4.10.18   True        False         False      16d     
kube-scheduler                             4.10.18   True        False         False      16d     
machine-api                                4.10.18   True        False         False      16d     
machine-approver                           4.10.18   True        False         False      16d     
machine-config                             4.10.18   True        False         False      16d     
marketplace                                4.10.18   True        False         False      16d     
network                                    4.10.18   True        False         False      16d     
node-tuning                                4.10.18   True        False         False      11h     
openshift-apiserver                        4.10.18   True        False         False      11h     
openshift-controller-manager               4.10.18   True        False         False      11h     
openshift-samples                          4.10.18   True        False         False      16d     
operator-lifecycle-manager                 4.10.18   True        False         False      16d     
operator-lifecycle-manager-catalog         4.10.18   True        False         False      16d     
operator-lifecycle-manager-packageserver   4.10.18   True        False         False      11h     
service-ca                                 4.10.18   True        False         False      16d     
2022.07.07 22:15:49 [userid@host ~/git/osecli/ocp_resources] $ oc project | grep backup
Using project "backup" on server "https://api.crc.testing:6443".
2022.07.07 22:15:57 [userid@host ~/git/osecli/ocp_resources] $ oc get pods -n openshift-image-registry
NAME                                              READY   STATUS      RESTARTS   AGE
cluster-image-registry-operator-5959746b9-4tq76   1/1     Running     0          15d
image-pruner-27620640-d76p8                       0/1     Completed   0          3h16m
image-registry-7dc8f48898-8jztf                   1/1     Running     0          11h
node-ca-kkrfz                                     1/1     Running     0          16d
2022.07.07 22:16:12 [userid@host ~/git/osecli/ocp_resources] $ oc logs -n openshift-image-registry image-registry-7dc8f48898-8jztf | tail
time="2022-07-08T03:16:07.345738576Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=173c9211-da64-402b-8521-c8576d233530 http.request.method=GET http.request.remoteaddr="10.217.0.1:54492" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="23.673µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:07.345763881Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=85519578-d484-4de0-ac84-e8daf9087b33 http.request.method=GET http.request.remoteaddr="10.217.0.1:54490" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="16.355µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:17.347388073Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=976636b3-c65b-4157-810a-02630f616c73 http.request.method=GET http.request.remoteaddr="10.217.0.1:54676" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="25.159µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:17.347683133Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=2b754824-c390-443f-b821-f8ee15db4f89 http.request.method=GET http.request.remoteaddr="10.217.0.1:54674" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="13.388µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:27.345786456Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=44b3bc60-1f84-4153-9aaf-9684a3b9c95e http.request.method=GET http.request.remoteaddr="10.217.0.1:54878" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="28.977µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:27.346075778Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=187b76db-4be2-4ba9-97e0-4af9fc8a443f http.request.method=GET http.request.remoteaddr="10.217.0.1:54880" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="16.331µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:37.347688518Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=7d45817d-05f9-4f5e-98d4-468457bfeca7 http.request.method=GET http.request.remoteaddr="10.217.0.1:55092" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="25.419µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:37.348086428Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=fe2f5a8c-408b-4a76-8bdc-979d5eb91838 http.request.method=GET http.request.remoteaddr="10.217.0.1:55094" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="18.906µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:47.348250833Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=b11ec538-5704-4007-ad6b-b044e55a9822 http.request.method=GET http.request.remoteaddr="10.217.0.1:55284" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="50.495µs" http.response.status=200 http.response.written=0
time="2022-07-08T03:16:47.348512699Z" level=info msg=response go.version=go1.17.5 http.request.host="10.217.0.57:5000" http.request.id=14cd2483-80f1-47be-952c-f7488cf9190a http.request.method=GET http.request.remoteaddr="10.217.0.1:55282" http.request.uri=/healthz http.request.useragent=kube-probe/1.23 http.response.duration="16.311µs" http.response.status=200 http.response.written=0

I'm wondering if it has something to do with net namespaces? Something similar to the docker on osx issue interacting via namespaces?

[ayates83]

Something similar to this issue?
#2354

[ayates83]

Fails with same messages on the latest release as well:

 $ crc version

CRC version: 2.6.0+e7251680
OpenShift version: 4.10.22
Podman version: 4.1.0


$ podman push localhost/ocpbackup $REG/backup/ocpbackup

Error: error pushing image "default-route-openshift-image-registry.apps-crc.testing/backup/ocpbackup": trying to reuse blob sha256:5bc03dec623975972bc748cb1da06ce8b4cf854d38c854a9e5f5bcad48caf6cc at destination: pinging container registry default-route-openshift-image-registry.apps-crc.testing: Get "https://default-route-openshift-image-registry.apps-crc.testing/v2/": dial tcp 127.0.0.1:443: connect: connection refused

I can connect to the URL using curl:

 $ echo $REG
default-route-openshift-image-registry.apps-crc.testing

$ curl -kv https://$REG
*   Trying 127.0.0.1:443...
* Connected to default-route-openshift-image-registry.apps-crc.testing (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.apps-crc.testing
*  start date: Jul  6 12:45:45 2022 GMT
*  expire date: Jul  5 12:45:46 2024 GMT
*  issuer: CN=ingress-operator@1657111544
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET / HTTP/1.1
> Host: default-route-openshift-image-registry.apps-crc.testing
> User-Agent: curl/7.79.1
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< cache-control: no-cache
< date: Wed, 27 Jul 2022 11:53:34 GMT
< content-length: 0
< set-cookie: 34727b82525eb26a530629c5bf0ec2f2=4b9996899c7e3a089f410965ad2bfd8a; path=/; HttpOnly; Secure; SameSite=None
< 
* Connection #0 to host default-route-openshift-image-registry.apps-crc.testing left intact

I can hit the v2 api endpoint with a bearer token and list the existing images:

$ curl -kv -H "Authorization: Bearer ${TOK}" https://$REG/v2/_catalog
*   Trying 127.0.0.1:443...
* Connected to default-route-openshift-image-registry.apps-crc.testing (127.0.0.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=*.apps-crc.testing
*  start date: Jul  6 12:45:45 2022 GMT
*  expire date: Jul  5 12:45:46 2024 GMT
*  issuer: CN=ingress-operator@1657111544
*  SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
> GET /v2/_catalog HTTP/1.1
> Host: default-route-openshift-image-registry.apps-crc.testing
> User-Agent: curl/7.79.1
> Accept: */*
> Authorization: Bearer sha256~A0M149SEElavwCbp7BBXkMF1ieBQ4YUzUyT0NH7L_zo
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json; charset=utf-8
< docker-distribution-api-version: registry/2.0
< x-registry-supports-signatures: 1
< date: Wed, 27 Jul 2022 12:04:46 GMT
< transfer-encoding: chunked
< set-cookie: 34727b82525eb26a530629c5bf0ec2f2=4b9996899c7e3a089f410965ad2bfd8a; path=/; HttpOnly; Secure; SameSite=None
< cache-control: private
< 
{"repositories":["openshift/apicast-gateway","openshift/apicurito-ui","openshift/cli","openshift/cli-artifacts","openshift/dotnet","openshift/dotnet-runtime","openshift/driver-toolkit","openshift/fis-java-openshift","openshift/fis-karaf-openshift","openshift/fuse-apicurito-generator","openshift/fuse7-console","openshift/fuse7-eap-openshift","openshift/fuse7-eap-openshift-java11","openshift/fuse7-java-openshift","openshift/fuse7-java11-openshift","openshift/fuse7-karaf-openshift","openshift/fuse7-karaf-openshift-jdk11","openshift/golang","openshift/httpd","openshift/installer","openshift/installer-artifacts","openshift/java","openshift/java-runtime","openshift/jboss-amq-62","openshift/jboss-amq-63","openshift/jboss-datagrid65-client-openshift","openshift/jboss-datagrid65-openshift","openshift/jboss-datagrid71-client-openshift","openshift/jboss-datagrid71-openshift","openshift/jboss-datagrid72-openshift","openshift/jboss-datagrid73-openshift","openshift/jboss-datavirt64-driver-openshift","openshift/jboss-datavirt64-openshift","openshift/jboss-decisionserver64-openshift","openshift/jboss-eap-xp3-openjdk11-openshift","openshift/jboss-eap-xp3-openjdk11-runtime-openshift","openshift/jboss-eap64-openshift","openshift/jboss-eap70-openshift","openshift/jboss-eap71-openshift","openshift/jboss-eap72-openjdk11-openshift-rhel8","openshift/jboss-eap72-openshift","openshift/jboss-eap73-openjdk11-openshift","openshift/jboss-eap73-openjdk11-runtime-openshift","openshift/jboss-eap73-openshift","openshift/jboss-eap73-runtime-openshift","openshift/jboss-eap74-openjdk11-openshift","openshift/jboss-eap74-openjdk11-runtime-openshift","openshift/jboss-eap74-openjdk8-openshift","openshift/jboss-eap74-openjdk8-runtime-openshift","openshift/jboss-fuse70-console","openshift/jboss-fuse70-eap-openshift","openshift/jboss-fuse70-java-openshift","openshift/jboss-fuse70-karaf-openshift","openshift/jboss-processserver64-openshift","openshift/jboss-webserver31-tomcat7-openshift","openshift/jboss-webserver31-tomcat8-openshift","openshift/jboss-webserver56-openjdk11-tomcat9-openshift-ubi8","openshift/jboss-webserver56-openjdk8-tomcat9-openshift-ubi8","openshift/jenkins","openshift/jenkins-agent-base","openshift/jenkins-agent-maven","openshift/jenkins-agent-nodejs","openshift/mariadb","openshift/must-gather","openshift/mysql","openshift/nginx","openshift/nodejs","openshift/oauth-proxy","openshift/openjdk-11-rhel7","openshift/openjdk-11-rhel8","openshift/perl","openshift/php","openshift/postgresql","openshift/python","openshift/redhat-openjdk18-openshift","openshift/redhat-sso70-openshift","openshift/redhat-sso71-openshift","openshift/redhat-sso72-openshift","openshift/redhat-sso73-openshift","openshift/redis","openshift/rhdm-decisioncentral-rhel8","openshift/rhdm-kieserver-rhel8","openshift/rhpam-businesscentral-monitoring-rhel8","openshift/rhpam-businesscentral-rhel8","openshift/rhpam-kieserver-rhel8","openshift/rhpam-smartrouter-rhel8","openshift/ruby","openshift/sso74-openshift-rhel8","openshift/sso75-openshift-rhel8","openshift/tests","openshift/tools","openshift/ubi8-openjdk-11","openshift/ubi8-openjdk-11-runtime","openshift/ubi8-openjdk-8","openshift/ubi8-openjdk-8-runtime"]}
* Connection #0 to host default-route-openshift-image-registry.apps-crc.testing left intact

[ayates83]

I also tried forwarding port 5000 from the internal registry pod to my localhost and pushing there, but failed as well:

 $ podman push localhost/ocpbackup 127.0.0.1:5000/openshift/ocpbackup
Error: error pushing image "127.0.0.1:5000/openshift/ocpbackup": trying to reuse blob sha256:5bc03dec623975972bc748cb1da06ce8b4cf854d38c854a9e5f5bcad48caf6cc at destination: pinging container registry 127.0.0.1:5000: Get "https://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused

[praveenkumar]

I also tried forwarding port 5000 from the internal registry pod to my localhost and pushing there, but failed as well:

 $ podman push localhost/ocpbackup 127.0.0.1:5000/openshift/ocpbackup
Error: error pushing image "127.0.0.1:5000/openshift/ocpbackup": trying to reuse blob sha256:5bc03dec623975972bc748cb1da06ce8b4cf854d38c854a9e5f5bcad48caf6cc at destination: pinging container registry 127.0.0.1:5000: Get "https://127.0.0.1:5000/v2/": dial tcp 127.0.0.1:5000: connect: connection refused

@ayates83 127.0.0.1:5000 is not going to work because 5000 port is only internally available not externally. Also are you using podman-machine for podman which is running parallel to crc?

[ayates83]

I'm running the podman that's included with crc:

 $ which podman
/Users/$user/.crc/bin/oc/podman

$ podman --version
podman version 3.2.3

I removed podman provided by brew (was using it previously.) But it looks like my podman is pointing to the one included with crc.

[praveenkumar]

I'm running the podman that's included with crc:

 $ which podman
/Users/$user/.crc/bin/oc/podman

$ podman --version
podman version 3.2.3

I removed podman provided by brew (was using it previously.) But it looks like my podman is pointing to the one included with crc.

Do you have docker desktop installed in your system?

[ayates83]

I did at one point but deleted it.

[praveenkumar]

@ayates83 I don't know why it is happening but one last debug you can do is ssh to the VM and then try to push the image to registry and see if that works. https://github.com/code-ready/crc/wiki/Debugging-guide have way to ssh to the VM.