Issue with containerd image pulling from non-compliant registries

[GlebBeloded]

Hello everyone and thank you for you work on this project!

This issue is not directly related to k3d, k3s, not even containerd is to be blamed, but I thought that this would be a good place to ask for guidance.

I've been using k3d for local development, and it was working perfectly fine until we migrated to github enterprise. As far as I can understand, github registry is not fully OCI compliant. While trying to pull an image from registry, I get following error:
Warning Failed 22s (x3 over 64s) kubelet Failed to pull image "foo.bar/svc/app/app:master": rpc error: code = NotFound desc = failed to pull and unpack image "foo.bar/svc/app/app:master:master": failed to copy: httpReaderSeeker: failed open: content at https://docker.foo.bar/svc/app/app:master/manifests/sha256:493329b0f6b9004d37b6737e0eddb42bdde887b6c69bf6eff47a536b2c3c5be4 not found: not found

I've been trying to find workarounds for this, for example running this proxy, however it also seems not be fully OCI compliant, since i receive another error:
containerd failed to unpack image on snapshotter overlayfs: unexpected media type: application/octet-stream
As i understand the octet-stream header was initially supported by containerd, but support for it was removed some time ago.

I was unable to get into black magic wizardry of launching k3s with docker container runtime, and I was not able to find a working proxy which would be "good-enough" for containerd.

It is obvious that github packages developers should be making their api OCI compliant, however, while they are at it, kubernetes is now using containerd by default, and even docker tells me that api that the registry uses is deprecated, surely I am missing something obvious here?
Thank you again and have a great day!

[iwilltry42]

Hi @GlebBeloded , thanks for starting this discussion!
Even though it may not be directly related to k3d, I still think it's an important piece to think about.

I just gave it a try and it worked fine for me with this test:

• Push an image to ghcr.io: docker push ghcr.io/iwilltry42/docker-nginx-php:1.20.0-8.0.5
• Create k3d cluster: k3d cluster create -a 1
• Exec into k3d cluster: docker exec -it k3d-k3s-default-agent-0
• Ask containerd to pull that image: ctr --debug image pull ghcr.io/iwilltry42/docker-nginx-php:1.20.0-8.0.5 -u iwilltry42:xyz
• Confirm that image was pulled successfully

So to me this looks like there's no problem with containerd pulling from ghcr.
Or is this maybe about the legacy Github docker registry? Since you show docker.foo.bar in your question, is it maybe GitHub Enterprise? 🤔

[GlebBeloded]

It is github enterprise, sorry for not mentioning that!
I believe that the version of ghcr they are using in cloud and enterprise might be quite different.
I wish there was an endpoint telling me what registry I am using, but as far as I know there is no way to tell ask registry about it's kind or version.

[iwilltry42]

Not sure if you can do it via the API or anything and I also don't know, if the URL would change in an Enterprise setup, but maybe just checking for the logo in the Web-UI is simple and good enough to see if the images are in the new package registry already: https://docs.github.com/en/packages/working-with-a-github-packages-registry/migrating-to-the-container-registry-from-the-docker-registry#how-can-you-tell-if-your-images-have-been-migrated 🤔