Skip to content
This repository has been archived by the owner on Nov 30, 2021. It is now read-only.

Failed to push some refs #826

Open
Overdrivr opened this issue Jun 14, 2017 · 20 comments
Open

Failed to push some refs #826

Overdrivr opened this issue Jun 14, 2017 · 20 comments

Comments

@Overdrivr
Copy link

I'm trying to deploy from Gitlab CI to Deis using buildpack ( I cannot use Docker-based because of #823 )

Authentication to deis from gitlab-ci runner works fine, code is built successfully. But at the end of the deployment, I get the following error message.

remote: Successfully built 76385f485fae        
remote: Pushing to registry        
remote: received unexpected HTTP status: 500 Internal Server Error        
remote: 2017/06/14 06:09:56 Error running git receive hook [Build pod exited with code 1, stopping build.]        
To ssh://[email protected]:2222/cronobo-dispatcher.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'ssh://[email protected]:2222/cronobo-dispatcher.git'
ERROR: Job failed: exit code 1

My cluster is GKE 3 node n1-standard1, with 50Go of storage each.

All pods are properly running

$ kubectl --namespace=deis get pods
NAME                                     READY     STATUS    RESTARTS   AGE
deis-builder-1134541883-4mvsv            1/1       Running   0          2h
deis-controller-2381495828-61b1t         1/1       Running   1          2h
deis-database-401209816-csdkz            1/1       Running   0          2h
deis-logger-2717637750-16mtl             1/1       Running   3          2h
deis-logger-fluentd-4n8b3                1/1       Running   0          2h
deis-logger-fluentd-86dsg                1/1       Running   0          2h
deis-logger-fluentd-x1w8f                1/1       Running   0          2h
deis-logger-redis-1413683677-vl4kd       1/1       Running   0          2h
deis-minio-3370481340-8nlfx              1/1       Running   0          2h
deis-monitor-grafana-1073006293-45rnw    1/1       Running   0          2h
deis-monitor-influxdb-2675149720-vmp7s   1/1       Running   0          2h
deis-monitor-telegraf-n9z5j              1/1       Running   1          2h
deis-monitor-telegraf-qhbtf              1/1       Running   1          2h
deis-monitor-telegraf-ttsgr              1/1       Running   1          2h
deis-nsqd-1225249577-c3jnk               1/1       Running   0          2h
deis-registry-2623437609-x1fcp           1/1       Running   1          2h
deis-registry-proxy-3mwz3                1/1       Running   0          2h
deis-registry-proxy-3pftk                1/1       Running   0          2h
deis-registry-proxy-ccqxp                1/1       Running   0          2h
deis-router-3258585802-fmfm9             1/1       Running   0          2h
deis-workflow-manager-3757950027-vkwn8   1/1       Running   0          2h

How can I debug this ?
@dumyan
Copy link

dumyan commented Jun 14, 2017

The push is rejected because of the Internal Server Error that happened probably when pushing the built image into registry (I'm not sure if a successful push to registry is logged). This is desired behavior because if the refs get pushed and the image cannot be successfully started in a container it will leave you with a bogus repository state which is not currently deployed. So, failed to push some refs is really your friend here and your enemies are probably hiding in the logs.

In short, you should inspect deis-builder and also deis-registry logs and work your way from there. You can also enable debug mode if you see nothing obvious in the logs.

@Overdrivr
Copy link
Author

Thanks for the reply. The registry pod if filled with info messages, is this the default behavior ?

10.12.0.1 - - [15/Jun/2017:05:53:12 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-06-15T05:53:12.050694806Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="10.11.0.12:5000" http.request.id=fe682b3f-5756-47d9-8e86-cf19f5d1a9e3 http.request.method=GET http.request.remoteaddr="10.12.0.1:35296" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=8.927239ms http.response.status=200 http.response.written=2 instance.id=dd03c229-fbfe-4309-95c3-3667e7847fe1 service=registry version=v2.6.0 
10.12.0.1 - - [15/Jun/2017:05:53:12 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-06-15T05:53:22.041218766Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="10.11.0.12:5000" http.request.id=710364ac-dde4-4bb2-9723-cd2d383ca686 http.request.method=GET http.request.remoteaddr="10.12.0.1:35330" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.648287ms http.response.status=200 http.response.written=2 instance.id=dd03c229-fbfe-4309-95c3-3667e7847fe1 service=registry version=v2.6.0 
10.12.0.1 - - [15/Jun/2017:05:53:22 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-06-15T05:53:22.04325269Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="10.11.0.12:5000" http.request.id=80e1cbd3-52e5-405c-ab1c-928957f731de http.request.method=GET http.request.remoteaddr="10.12.0.1:35328" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" http.response.contenttype="application/json; charset=utf-8" http.response.duration=1.529342ms http.response.status=200 http.response.written=2 instance.id=dd03c229-fbfe-4309-95c3-3667e7847fe1 service=registry version=v2.6.0

The deis-builder log is maybe more interesting

$ kubectl --namespace=deis logs deis-builder-1134541883-4mvsv 
2017/06/14 04:37:38 Starting health check server on port 8092
2017/06/14 04:37:38 Starting deleted app cleaner
2017/06/14 04:37:38 Starting SSH server on 0.0.0.0:2223
Listening on 0.0.0.0:2223
Accepting new connections.
---> ---> 2017/06/14 04:37:51 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:01 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:11 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:21 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:31 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:41 Readinesscheck endpoint timed out after 10s
2017/06/14 04:38:51 Readinesscheck endpoint timed out after 10s
2017/06/14 04:39:01 Readinesscheck endpoint timed out after 10s
Accepted connection.
Starting ssh authentication
Channel type: session

receiving git repo name: app-dispatcher.git, operation: git-receive-pack, fingerprint: 84:a8:7e:6a:e8:1b:20:34:4b:29:a7:f3:14:5a:e3:42, user: gitlabci
creating repo directory /home/git/app-dispatcher.git
writing pre-receive hook under /home/git/app-dispatcher.git
git-shell -c git-receive-pack 'app-dispatcher.git'
Waiting for git-receive to run.
Waiting for deploy.
Deploy complete.
Accepted connection.
Starting ssh authentication
Channel type: session

receiving git repo name: app-dispatcher.git, operation: git-receive-pack, fingerprint: 84:a8:7e:6a:e8:1b:20:34:4b:29:a7:f3:14:5a:e3:42, user: gitlabci
creating repo directory /home/git/app-dispatcher.git
writing pre-receive hook under /home/git/app-dispatcher.git
git-shell -c git-receive-pack 'app-dispatcher.git'
Waiting for git-receive to run.
Waiting for deploy.
Deploy complete.
Accepted connection.
---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> ---> [ERROR] Failed handshake: EOF
Accepted connection.
Accepted connection.
---> ---> [ERROR] Failed handshake: EOF
[ERROR] Failed handshake: EOF
Accepted connection.
Accepted connection.
---> ---> [ERROR] Failed handshake: EOF
Accepted connection.
---> [ERROR] Failed handshake: EOF

Any idea what is causing the handshake to fail ?

@bacongobbler
Copy link
Member

This handshake failures are just the healthcheck probe. It's a red herring. What you want to be looking at is the registry logs as that's where the 500 error came from.

@Overdrivr
Copy link
Author

Here you go, registry logs with HTTP 500 error

$ kubectl --namespace=deis logs deis-registry-2623437609-x1fcp | grep " 500 "

10.12.0.7 - - [15/Jun/2017:06:15:15 +0000] "PUT /v2/app-dispatcher/blobs/uploads/a0a37674-7680-4e64-a809-05551e124f76?_state=9rUDYtxw_nxrKtVQVQK20EXSMhdleDG4Ci5BbZEZOjt7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiYTBhMzc2NzQtNzY4MC00ZTY0LWE4MDktMDU1NTFlMTI0Zjc2IiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNDozOFoifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:20 +0000] "PUT /v2/app-dispatcher/blobs/uploads/05e4cd44-be93-459c-99a4-cf7d77f6519a?_state=omTy6YEGZqDnBs2fqLPOXwWbg6PTtpBzGAh8eVtsIMh7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMDVlNGNkNDQtYmU5My00NTljLTk5YTQtY2Y3ZDc3ZjY1MTlhIiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNDozOVoifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:54 +0000] "PUT /v2/app-dispatcher/blobs/uploads/f1cb1743-43c3-478b-a4b8-27c4ad3e4a8b?_state=xhjHl05OxmV09G4oskfU0CIC7gsbaczC5Evz1iX36jx7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiZjFjYjE3NDMtNDNjMy00NzhiLWE0YjgtMjdjNGFkM2U0YThiIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNToyMVoifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:55 +0000] "PUT /v2/app-dispatcher/blobs/uploads/d3fbfd1a-eb57-4cf8-a55b-fb26e0d38ac6?_state=qd0QZx4wanTPwiDZ7aVnXpr1vxR7fMZ_gHTEdmC8b1N7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiZDNmYmZkMWEtZWI1Ny00Y2Y4LWE1NWItZmIyNmUwZDM4YWM2IiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNToyNloifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:15:56 +0000] "PUT /v2/app-dispatcher/blobs/uploads/2a7e85ff-768e-4407-b2e2-810261e0e5d7?_state=2mDRWSFFneGaJL4styC3IQ7Cd1bjnpw5FcOUZpiLUNB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMmE3ZTg1ZmYtNzY4ZS00NDA3LWIyZTItODEwMjYxZTBlNWQ3IiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTQ6MzhaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:16:47 +0000] "PUT /v2/app-dispatcher/blobs/uploads/4213afc5-bd0c-4e89-8545-c7c62d97c973?_state=33N3ehczUMDazWB9sxMH7wN8ftYcLOL4hWyps1L_3P17Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNDIxM2FmYzUtYmQwYy00ZTg5LTg1NDUtYzdjNjJkOTdjOTczIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNjowNVoifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:16:48 +0000] "PUT /v2/app-dispatcher/blobs/uploads/2670a319-80f6-4b31-9c60-8498fb2219c0?_state=wAgGDjU4XQc-K3tBivJ11HF0bi2rgxXNaocIxf5T9pB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMjY3MGEzMTktODBmNi00YjMxLTljNjAtODQ5OGZiMjIxOWMwIiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNjowNloifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:16:55 +0000] "PUT /v2/app-dispatcher/blobs/uploads/46b17962-9a71-4e45-ae03-23b207184548?_state=aunuxweYQROhndJe4ZqAsmnCBLyWbq6r1P-DKY1p-nR7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNDZiMTc5NjItOWE3MS00ZTQ1LWFlMDMtMjNiMjA3MTg0NTQ4IiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTY6MDJaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:17:36 +0000] "PUT /v2/app-dispatcher/blobs/uploads/3a644e41-d3c2-44a8-9596-ba42359aa71c?_state=1WfuVKBc2gc3qjb0siH7Lcbf2asHkHjdBDjYOh8T_ZB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiM2E2NDRlNDEtZDNjMi00NGE4LTk1OTYtYmE0MjM1OWFhNzFjIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNzowMloifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:17:44 +0000] "PUT /v2/app-dispatcher/blobs/uploads/787c7538-1a4e-439c-ad51-a0658ef87e74?_state=b7y6Wvq1uz0pKW1PluUWMABckM3NtIRP6X9zu0IghcZ7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNzg3Yzc1MzgtMWE0ZS00MzljLWFkNTEtYTA2NThlZjg3ZTc0IiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNzowNFoifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:17:55 +0000] "PUT /v2/app-dispatcher/blobs/uploads/4b5c16e3-2c1e-4264-a6f4-8cc396affbac?_state=vM16auRY8mezk2pPhMIHKLIzgzVhjV43KRhxQihhYid7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiNGI1YzE2ZTMtMmMxZS00MjY0LWE2ZjQtOGNjMzk2YWZmYmFjIiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTc6MDVaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:18:14 +0000] "PUT /v2/app-dispatcher/blobs/uploads/b3928bcf-36e5-4000-886c-7288fe3389c0?_state=Uhwuv4nSaQCI_MUn8Eeqc8q8OzO1KM74-_YwRZ7JER57Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiYjM5MjhiY2YtMzZlNS00MDAwLTg4NmMtNzI4OGZlMzM4OWMwIiwiT2Zmc2V0Ijo0MzIyNTU2NywiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxNzo1N1oifQ%3D%3D&digest=sha256%3A4f38f9d5c3c0e3e4eabe6d4dac4bcc729caf847c172d0af75caf581b6c23120c HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:18:38 +0000] "PUT /v2/app-dispatcher/blobs/uploads/106c9553-86b6-4dfa-8b22-14bbf712b80f?_state=-5FSF76b4zMHCcWh2qKOfujnGVMTYZYp8dC7c73EFuB7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMTA2Yzk1NTMtODZiNi00ZGZhLThiMjItMTRiYmY3MTJiODBmIiwiT2Zmc2V0Ijo1MjU2ODI2OSwiU3RhcnRlZEF0IjoiMjAxNy0wNi0xNVQwNjoxODowNVoifQ%3D%3D&digest=sha256%3Aef0380f84d05d3cdc5a5f660232d35ccb020ccf1d635b585580dea44691a13a7 HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:18:49 +0000] "PUT /v2/app-dispatcher/blobs/uploads/f5c80612-d1a4-44b7-9f1a-75a6096718ed?_state=BS_ZRX1Sv0kdnvr25YbNzOG8-f7jjJef2aTRlFKWf317Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiZjVjODA2MTItZDFhNC00NGI3LTlmMWEtNzVhNjA5NjcxOGVkIiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTg6MTBaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
10.12.0.7 - - [15/Jun/2017:06:19:38 +0000] "PUT /v2/app-dispatcher/blobs/uploads/0592b1a6-5234-458d-9c8a-fa324a050517?_state=iNYDPE62ScgvCj825K8D-xthNqd_YffLVV2JfTzoQRd7Ik5hbWUiOiJjcm9ub2JvLWRpc3BhdGNoZXIiLCJVVUlEIjoiMDU5MmIxYTYtNTIzNC00NThkLTljOGEtZmEzMjRhMDUwNTE3IiwiT2Zmc2V0IjoxMzE4MTkyMDcsIlN0YXJ0ZWRBdCI6IjIwMTctMDYtMTVUMDY6MTk6MDlaIn0%3D&digest=sha256%3A4125326b53d8463d437a9052d8fb892e1bf9e7dc8d68518ac9780611476123ec HTTP/1.0" 500 117 "" "docker/1.11.2 go/go1.7.4 git-commit/4dc5990 kernel/4.4.35+ os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"

@bacongobbler
Copy link
Member

Sounds like it's having a hard time uploading images to the backing storage driver as it's barking during PUT operations. I'd check your object storage logs and see if it's crashing, or perhaps your cluster is misconfigured and the registry is unable to upload blobs to object storage.

@Overdrivr
Copy link
Author 

$ kubectl --namespace=deis logs deis-minio-3370481340-8nlfx 

minio server /home/minio/
2017/06/14 04:37:02 starting Minio server
2017/06/14 04:37:02 starting health check server on 0.0.0.0:8082
Migration from version ‘2’ to ‘3’ completed successfully.
Migration from version ‘3’ to ‘4’ completed successfully.

AccessKey: ****************  SecretKey: ********************  Region: us-east-1

Minio Object Storage:
    http://127.0.0.1:9000
    http://**.**.**.**:9000

Minio Browser:
    http://127.0.0.1:9000
    http://**.**.**.**:9000

To configure Minio Client:
    $ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
    $ chmod 755 mc
    $ ./mc config host add myminio http://127.0.0.1:9000 ****************   *************************

It's weird that minio is using region us-east-1, because my cluster is located in europe-west1-b. Could this be the problem ?

@bacongobbler
Copy link
Member

Minio fakes out and simulates that it's in us-east-1 to mimic the S3 API. It doesn't accurately reflect where it's deployed (in the cluster).

@EamonZhang
Copy link

+1

@ryane ryane mentioned this issue Jul 21, 2017
Closed
@wikkid
Copy link

wikkid commented Aug 7, 2017 •
edited
Loading

Any updates on this? Got exactly the same problem.
I have bare metal kubernetes cluster and Deis workflow on top of that.
Sample applications are fine, but it doesn't work with my own.

@robeferre
Copy link

Hi Guys any update on this Im having the same issue

@bacongobbler bacongobbler mentioned this issue Aug 8, 2017
Closed
@Overdrivr
Copy link
Author

I wasn't sure how to debug this further, so no updates so far. I wasn't seeing anything in my object storage logs.

@galbacarys
Copy link

galbacarys commented Aug 18, 2017 •
edited
Loading

Having watched this carefully (I know, great way to debug things) I noticed something interesting-It looks like some of the minio requests are going through, but then the connection gets dropped somewhere (the object is too big?) and minio barfs and stops taking any more data.

[ deis registry logs ]
100.96.6.18 - - [18/Aug/2017:16:40:25 +0000] "POST /v2/aws-dx-selfjoin/blobs/uploads/ HTTP/1.0" 202 0 "" "docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
time="2017-08-18T16:40:27.09674817Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="100.96.6.23:5000" http.request.id=cd874d81-d36b-48ec-8455-4528504f88f8 http.request.method=GET http.request.remoteaddr="100.96.6.1:53204" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" h
ttp.response.contenttype="application/json; charset=utf-8" http.response.duration=1.746968ms http.response.status=200 http.response.written=2 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry version=v2.6.0
100.96.6.1 - - [18/Aug/2017:16:40:27 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-08-18T16:40:27.09757101Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="100.96.6.23:5000" http.request.id=72f0a983-963a-499e-89d6-d30c1b29fab7 http.request.method=GET http.request.remoteaddr="100.96.6.1:53202" http.request.uri="/v2/" http.request.useragent="Go-http-client/1.1" h
ttp.response.contenttype="application/json; charset=utf-8" http.response.duration=1.58666ms http.response.status=200 http.response.written=2 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry version=v2.6.0
100.96.6.1 - - [18/Aug/2017:16:40:27 +0000] "GET /v2/ HTTP/1.1" 200 2 "" "Go-http-client/1.1"
time="2017-08-18T16:40:35.222805315Z" level=info msg="response completed" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=6aaf4e8a-dc20-4dcb-a9da-9f2df76db945 http.request.method=PATCH http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-
9ab733dd225f?_state=0CLtxmCoH9cYWnDAEmjo9mWBMC3_Q7TX85-MY80Au4B7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDE2OjQwOjI1LjIxODg4NzY5MVoifQ%3D%3D" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4
.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" http.response.duration=1.380391587s http.response.status=202 http.response.written=0 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry version=v2.6.0
100.96.6.18 - - [18/Aug/2017:16:40:33 +0000] "PATCH /v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=0CLtxmCoH9cYWnDAEmjo9mWBMC3_Q7TX85-MY80Au4B7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0IjowLCJTdGFydGVkQXQiOiIyMDE3LTA4LTE4VDE2
OjQwOjI1LjIxODg4NzY5MVoifQ%3D%3D HTTP/1.0" 202 0 "" "docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"
time="2017-08-18T16:40:35.378123268Z" level=error msg="unknown error completing upload: s3aws: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: DZFQC2IC58RHYLEY" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=c3edb847-d84f-4394-b247-0d03d594ce4f http.request.method=
PUT http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjA
xNy0wOC0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f930bec5e854b80e312e7b3bdd0907845" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=regis
try vars.name=aws-dx-selfjoin vars.uuid=dfbcfd73-21a3-43bd-b51b-9ab733dd225f version=v2.6.0
time="2017-08-18T16:40:35.378179784Z" level=error msg="error canceling upload after error: already closed" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=c3edb847-d84f-4394-b247-0d03d594ce4f http.request.method=PUT http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/u
ploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjAxNy0wOC0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f93
0bec5e854b80e312e7b3bdd0907845" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry vars.name=aws-dx-selfjoin vars.uuid=dfbcfd73-21a3-43bd-b51b-9ab733dd225f version
=v2.6.0
time="2017-08-18T16:40:35.383839588Z" level=error msg="response completed with error" err.code=unknown err.detail="s3aws: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: DZFQC2IC58RHYLEY" err.message="unknown error" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=c3
edb847-d84f-4394-b247-0d03d594ce4f http.request.method=PUT http.request.remoteaddr=100.96.6.1 http.request.uri="/v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMz
ZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjAxNy0wOC0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f930bec5e854b80e312e7b3bdd0907845" http.request.useragent="docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" http.re
sponse.contenttype="application/json; charset=utf-8" http.response.duration=110.023913ms http.response.status=500 http.response.written=117 instance.id=bdee4072-d5d4-4f37-b206-f5555479cc40 service=registry vars.name=aws-dx-selfjoin vars.uuid=dfbcfd73-21a3-43bd-b51b-9ab733dd225f version=v2.6.0
100.96.6.18 - - [18/Aug/2017:16:40:35 +0000] "PUT /v2/aws-dx-selfjoin/blobs/uploads/dfbcfd73-21a3-43bd-b51b-9ab733dd225f?_state=V01hMgbvQsi-hTA76R9ljuV8B8En0ZGQNiIOy4i6d8N7Ik5hbWUiOiJhd3MtZHgtc2VsZmpvaW4iLCJVVUlEIjoiZGZiY2ZkNzMtMjFhMy00M2JkLWI1MWItOWFiNzMzZGQyMjVmIiwiT2Zmc2V0Ijo0NjIxMDUwOSwiU3RhcnRlZEF0IjoiMjAxNy0wOC
0xOFQxNjo0MDoyNVoifQ%3D%3D&digest=sha256%3A2df4c3c3bdb9ad0afa98766c03c7546f930bec5e854b80e312e7b3bdd0907845 HTTP/1.0" 500 117 "" "docker/1.12.6 go/go1.6.4 git-commit/78d1802 kernel/4.4.65-k8s os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)"

@dumyan
Copy link

dumyan commented Aug 18, 2017

Hey, guys, you can try with a newer minio image - just edit the minio deployment. Rereading this made me vaguely remember that I had some issues with the minio version that deis deploys by default.

@adam-stokes
Copy link

@dumyan What new minio image are you referring to?

@dumyan
Copy link

dumyan commented Aug 20, 2017

@battlemidget the OP is using minio as a storage backend for the registry. The default image for deis-minio is pretty outdated (or at least was outdated few months ago when I used minio as a backend storage). My suggestion is to change the minio image in the deis-minio deployment (you can try this image for example minio/minio:RELEASE.2017-07-24T18-27-35Z)

@adam-stokes
Copy link

@dumyan thanks im hitting this same issue, ill give that a try

@adam-stokes
Copy link

adam-stokes commented Aug 21, 2017 •
edited
Loading

@dumyan Ok, so not exactly the same issue as this one it seems. I am using off cluster storage (s3) and our distribution of kubernetes uses flannel cni so I just needed to enable that.

FYI, we are heavily promoting Deis workflow as part of our core addons for our Canonical Distribution of Kubernetes, my first blog post on it is here http://blog.astokes.org/conjure-up-dev-summary-aws-cloud-native-integration-and-vsphere-3/#addons, and the application is https://conjure-up.io. cc @bacongobbler

Thanks for the help!

@bacongobbler
Copy link
Member

@battlemidget you may want to read our latest blog post. https://deis.com/blog/2017/deis-workflow-final-release/

@adam-stokes
Copy link

@bacongobbler well that is disappointing :(

@Cryptophobia Cryptophobia mentioned this issue Mar 20, 2018
Open
@Cryptophobia
Copy link

This issue was moved to teamhephy/workflow#14

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@adam-stokes @Overdrivr @bacongobbler @galbacarys @dumyan @robeferre @EamonZhang @Cryptophobia @wikkid