Microk8s is not applying docker.io mirror changes

[whiskerch]

Hi,

I'm trying to change microk8s to use an internal mirror of docker hub to pull all images through, but it does not seem to apply the changes in the configuration I have made.

I am using a Sonatype/Nexus3 docker proxy to cache images we are using, and need to get microk8s to pull images from there instead of going direct to the https://registry-1.docker.io url

To do this I have modified the file at /var/snap/microk8s/current/args/containerd-template.toml and have changed this entry:

[plugins.cri.registry.mirrors."docker.io"]
    endpoint = ["https://registry-1.docker.io"]

to

[plugins.cri.registry.mirrors."docker.io"]
    endpoint = ["https://my-mirror.xxxxxxxx.com"]

I have then stopped and started the microk8s service (and can see the changes reflected in /var/snap/microk8s/current/args/containerd.toml

However, when I deploy Istio with the microk8s.enable istio command, and then look at the pods, I can see messages such as:

Failed to pull image "docker.io/istio/citadel/manifests/1.3.4": rpc error: code = Unknown desc = failed to resolve image "docker.io/istio/citadel/manifests/1.3.4": no available registry endpoint: failed to do request: Head "https://registry-1.docker.io/v2/istio/citadel/manifests/1.3.4" dial txp x.x.x.x:443: connect: connection refused

(I have blocked the docker.io on our infrastructure to make sure it was using the proxy)

However...

On the same box with docker I have this config in the daemon.json file:

{
    "registry-mirrors":["https://my-mirror.xxxxxxxx.com"]
}

If I run the command:

docker pull istio/citadel:1.3.4

It is able to use my mirror without any issue.

Is there anything I am missing when it comes to telling microk8s where to look for docker.io based images?

Please run microk8s inspect and attach the generated tarball to this issue.
inspection-report-20200828_124323.tar.gz

[whiskerch]

I think this issue could be the cause of my problems: containerd/containerd#4011

[hjanuschka]

same problem here.... can't get it to work.
need to get rid of docker.hub, because of the pull limit.
we run a harbor as mirror/proxy - but microk8s doesnt use it :'(

microk8s.ctr image pull  docker.io/argoproj/argocli:v2.11.7

still pulls from docker.io

[patsevanton]

Any news?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pweibert]

I still have this issue with MicroK8s v1.30.3 revision 7040
this is my /var/snap/microk8s/current/args/certs.d/docker.io/hosts.toml

server = "https://registry-1.docker.io"

[host."http://localhost:32000"]
  capabilities = ["pull", "resolve"]

I've also tried to apply a launch config in /var/snap/microk8s/common/.microk8s.yaml:

# microk8s-config.yaml
---
version: 0.1.0
addons:
  - name: dns
  - name: nvidia
#  - name: rbac
#  - name: ingress
#  - name: hostpath-storage
  - name: registry

# The extra arguments listed here are not required, as they would be set by the 'rbac' and the 'dns'
# addons respectively. However, we set them to save time by not having to restart the Kubernetes
# services a few times during cluster bootstrap.
extraKubeAPIServerArgs:
  --authorization-mode: RBAC,Node
extraKubeletArgs:
  --cluster-dns: 10.152.183.10
  --cluster-domain: cluster.local

containerdRegistryConfigs:
  # Use `http://my.proxy:5000` as a DockerHub registry mirror.
  docker.io: |
    [host."localhost:32000"]
    capabilities = ["pull", "resolve"]

This is what I get when I try to pull the image (I've disabled the internet access for my box in router settings, no connection to external network)

microk8s ctr images pull docker.io/postgres:10.4 --http-dump
INFO[0000] HEAD /v2/postgres/manifests/10.4 HTTP/1.1
INFO[0000] Host: registry-1.docker.io
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distrv1+json, /
INFO[0000] User-Agent: containerd/v1.6.28
INFO[0000]
INFO[0000] trying next host error="failed to do request: Head "https://server misbehaving" host=registry-1.docker.io
ctr: failed to resolve reference "docker.io/postgres:10.4": failed to do request: Head "https://regis misbehaving

While this does work indeed:
microk8s ctr images pull localhost:32000/postgres:10.4 --http-dump

[d-shehu]

I'm also having the same problem pweibert. Either the documentation is invalid or there is a bug. I'm current on v1.32.

https://microk8s.io/docs/dockerhub-limits

The proxy works perfectly with Docker using /etc/docker/daemon.json. I think this issue should be re-opened.