Failing to build image (Docker Rate Limit for jupyter/repo2docker)

[scottyhq]

kubectl describe pod -n prod build-pangeo-2ddata-2dpangeo-2ddocker-2dimages-b2b0c6-1c0-0b

  Warning  Failed          9m28s (x6 over 10m)  kubelet            Error: ImagePullBackOff
  Normal   Pulling         9m14s (x4 over 10m)  kubelet            Pulling image "jupyter/repo2docker:0.11.0-169.gae6c2dc"
  Warning  Failed          9m13s (x4 over 10m)  kubelet            Failed to pull image "jupyter/repo2docker:0.11.0-169.gae6c2dc": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
  Warning  Failed          9m13s (x4 over 10m)  kubelet            Error: ErrImagePull
  Normal   BackOff         30s (x45 over 10m)   kubelet            Back-off pulling image "jupyter/repo2docker:0.11.0-169.gae6c2dc"

[scottyhq]

According to https://www.docker.com/increase-rate-limits 'anonymous' pulls are limited to 100 per 6 hours.

I'm guessing the limit is tracked by IP address. I tried deleting a node, but the fresh node has the same errors, suggesting any node on the cluster appears as the same IP to DockerHub...

  Normal   Pulling    89s (x4 over 2m56s)  kubelet            Pulling image "docker:19.03.5-dind"
  Warning  Failed     88s (x4 over 2m55s)  kubelet            Failed to pull image "docker:19.03.5-dind": rpc error: code = Unknown desc = Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

pinging @sgibson91 @consideRatio in case they have run into this issue on other hubs / binder deployments and can suggest any workarounds (other than a paid DockerHub account :)

[sgibson91]

Hi @scottyhq. We don't run into this on mybinder.org because most of the clusters in the federation push built images to a private container registry, hence the number of images we pull from Docker Hub unauthenticated is very small. We also only pull images per node, zero pulls per launch.

You can pull repo2docker from quay.io which would help here https://quay.io/repository/jupyterhub/repo2docker

[minrk]

We did run into this for a while on OVH which had an admission controller that forces imagePullPolicy: Always and had to switch that federation member over to a private registry.

But the flagship GKE deployment ('prod' in mybinder.org-deploy) doesn't have any docker credentials and launches 1200 pods/hr. It pulls repo2docker from quay.io and pushes builds to GCR, but that's about it in terms of relevant config, I think.