The authentication for the Private Registry fails.

[shangjianan2]

The pod has been in the "ContainerCreating" state.

[root@dbSlave ~]# kubectl --kubeconfig=/home/wjl/dockerCompose/gitlab-runner-k3s/k3s/config get pods -n zbwt-prod -o wide
NAME                            READY   STATUS              RESTARTS   AGE   IP       NODE    NOMINATED NODE   READINESS GATES
app-k3s-test-555bddb6b9-bw7v2   0/1     ContainerCreating   0          34m   <none>   node3   <none>           <none>
[root@dbSlave ~]# 

The following logs are found in /var/lib/rancher/k3s/agent/containerd/containerd.log:

{"cniVersion":"1.0.0","forceAddress":true,"hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.42.2.0/24"}]],"routes":[{"dst":"10.42.0.0/16"}],"type":"host-local"},"isDefaultGateway":true,"isGateway":true,"mtu":1450,"name":"cbr0","type":"bridge"}time="2025-05-13T08:58:00.914367996+08:00" level=info msg="trying next host" error="pull access denied, repository does not exist or may require authorization: authorization failed: no basic auth credentials" host="10.111.160.21:39435"

Based on the above content, I analyze that the login username and password are incorrect.
The content of /etc/rancher/k3s/registries.yaml on the node is as follows:

mirrors:
  "*":
    endpoint:
      - "http://10.111.160.21:39435"
configs:
  "*":
    auth:
      username: admin
      password: Zbwt_123456

I restarted the service with the following command, to ensure that the k3s-agent loads the correct configuration.

systemctl restart k3s-agent

The login credentials (username and password) for the Private Registry have been verified to be correct on the node using Docker.

[root@node3 containerd]# docker login -u admin -p Zbwt_123456 10.111.160.21:39435
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[root@node3 containerd]# cat /root/.docker/config.json
{
	"auths": {
		"10.111.160.21:39435": {
			"auth": "YWRtaW46WmJ3dF8xMjM0NTY="
		}
	}
}[root@node3 containerd]# 

In summary, my question is why, when my username and password are correct (I can log in via Docker), there is an authentication failure error in the /var/lib/rancher/k3s/agent/containerd/containerd.log when the k3s-agent pulls the image?

[brandond]

Containerd does not respect the wildcard entry when authenticating, only when looking for mirrors.

If you want to use 10.111.160.21:39435 as a mirror for all registries, and it requires authentication, you should do:

mirrors:
  "*":
    endpoint:
      - "http://10.111.160.21:39435"
configs:
  10.111.160.21:39435:
    auth:
      username: admin
      password: Zbwt_123456

[shangjianan2]

Containerd does not respect the wildcard entry when authenticating, only when looking for mirrors.

If you want to use 10.111.160.21:39435 as a mirror for all registries, and it requires authentication, you should do:

mirrors:
"*":
endpoint:
- "http://10.111.160.21:39435"
configs:
10.111.160.21:39435:
auth:
username: admin
password: Zbwt_123456

Thank you very much. Your method worked. However, another problem has arisen.
The following logs are found in /var/lib/rancher/k3s/agent/containerd/containerd.log:

time="2025-05-13T09:47:27.059034225+08:00" level=info msg="PullImage \"10.111.160.21:39435/app-k3s-test-img:1747040141\""
time="2025-05-13T09:47:27.063040679+08:00" level=info msg="fetch failed" error="failed to do request: Head \"https://10.111.160.21:39435/v2/app-k3s-test-img/manifests/1747040141\": http: server gave HTTP response to HTTPS client" host="10.111.160.21:39435"
time="2025-05-13T09:47:27.064663343+08:00" level=error msg="PullImage \"10.111.160.21:39435/app-k3s-test-img:1747040141\" failed" error="rpc error: code = Unknown desc = failed to pull and unpack image \"10.111.160.21:39435/app-k3s-test-img:1747040141\": failed t
o resolve reference \"10.111.160.21:39435/app-k3s-test-img:1747040141\": failed to do request: Head \"https://10.111.160.21:39435/v2/app-k3s-test-img/manifests/1747040141\": http: server gave HTTP response to HTTPS client"
time="2025-05-13T09:47:27.064750049+08:00" level=info msg="stop pulling image 10.111.160.21:39435/app-k3s-test-img:1747040141: active requests=0, bytes read=0"
time="2025-05-13T09:47:41.775533733+08:00" level=info msg="PullImage \"10.111.160.21:39435/app-k3s-test-img:1747040141\""
time="2025-05-13T09:47:41.779586842+08:00" level=info msg="fetch failed" error="failed to do request: Head \"https://10.111.160.21:39435/v2/app-k3s-test-img/manifests/1747040141\": http: server gave HTTP response to HTTPS client" host="10.111.160.21:39435"
time="2025-05-13T09:47:41.781368217+08:00" level=error msg="PullImage \"10.111.160.21:39435/app-k3s-test-img:1747040141\" failed" error="rpc error: code = Unknown desc = failed to pull and unpack image \"10.111.160.21:39435/app-k3s-test-img:1747040141\": failed t
o resolve reference \"10.111.160.21:39435/app-k3s-test-img:1747040141\": failed to do request: Head \"https://10.111.160.21:39435/v2/app-k3s-test-img/manifests/1747040141\": http: server gave HTTP response to HTTPS client"
time="2025-05-13T09:47:41.781392979+08:00" level=info msg="stop pulling image 10.111.160.21:39435/app-k3s-test-img:1747040141: active requests=0, bytes read=0"

In the above logs, the message "http: server gave HTTP response to HTTPS client" is something I have encountered in Docker. I just need to configure "insecure-registries" in /etc/docker/daemon.json. Do I also need to make corresponding configurations in K3s?

[brandond]

No. The image app-k3s-test-img:1747040141 probably does not exist on your registry. When containerd gets an error from your registry (because the image does not exist), it falls back to the default endpoint, which uses https. You then get this error because it does not support https.

This is covered in the k3s docs: https://docs.k3s.io/installation/private-registry#default-endpoint-fallback

[shangjianan2]

No. The image app-k3s-test-img:1747040141 probably does not exist on your registry. When containerd gets an error from your registry (because the image does not exist), it falls back to the default endpoint, which uses https. You then get this error because it does not support https.

This is covered in the k3s docs: https://docs.k3s.io/installation/private-registry#default-endpoint-fallback

You said, "The image app-k3s-test-img:1747040141 probably does not exist on your registry", but this image actually exists because I can pull it using the following command

[root@node3 containerd]# docker rmi 10.111.160.21:39435/app-k3s-test-img:1747040141
Error response from daemon: No such image: 10.111.160.21:39435/app-k3s-test-img:1747040141
[root@node3 containerd]# docker pull 10.111.160.21:39435/app-k3s-test-img:1747040141
1747040141: Pulling from app-k3s-test-img
161d1c78b492: Pull complete 
efd79db3e1bc: Pull complete 
2275b5829f3b: Pull complete 
0231b03e491b: Pull complete 
9af3c3f95871: Pull complete 
229eb8fbf641: Pull complete 
3987c41e49c7: Pull complete 
bd9bec7914cf: Pull complete 
883ee90c3df7: Pull complete 
e9794ba36959: Pull complete 
b0579fe23395: Pull complete 
8d07b8af59bb: Pull complete 
f9658ff367fc: Pull complete 
Digest: sha256:69aa054425b82cee4fadff221f35ffb33695d8202e56efa67d5510c5294912e0
Status: Downloaded newer image for 10.111.160.21:39435/app-k3s-test-img:1747040141
10.111.160.21:39435/app-k3s-test-img:1747040141
[root@node3 containerd]# cat /etc/docker/daemon.json 
{
  "insecure-registries": ["10.111.160.21:39435"]
}
[root@node3 containerd]# 

[brandond]

Then something else is wrong. Check the containerd logs to see why the initial pull fails and it is trying again with https. There is nothing to fix on the K3s side. Only you have the tools to figure out what is going on with your environment, now that you know how containerd behaves.

[shangjianan2]

Then something else is wrong. Check the containerd logs to see why the initial pull fails and it is trying again with https. There is nothing to fix on the K3s side. Only you have the tools to figure out what is going on with your environment, now that you know how containerd behaves.

How to view the containerd logs? Is it through /var/lib/rancher/k3s/agent/containerd/containerd.log?

[brandond]

Yes. You can also set CONTAINERD_LOG_LEVEL=debug or CONTAINERD_LOG_LEVEL=trace in the k3s systemd unit's env vars if you need to turn up the verbosity.

[shangjianan2]

Yes. You can also set CONTAINERD_LOG_LEVEL=debug or CONTAINERD_LOG_LEVEL=trace in the k3s systemd unit's env vars if you need to turn up the verbosity.

I think I know what went wrong. It's the configuration of /etc/rancher/k3s/registries.yaml that's incorrect.

I use "CONTAINERD_LOG_LEVEL=trace" and find "time="2025-05-13T11:43:07.984225786+08:00" level=debug msg="loading host directory" dir="/var/lib/rancher/k3s/agent/etc/containerd/certs.d/10.111.160.21:39435". So I view /var/lib/rancher/k3s/agent/etc/containerd/certs.d/10.111.160.21:39435

# File generated by k3s. DO NOT EDIT.
server = "https://10.111.160.21:39435/v2"
capabilities = ["pull", "resolve", "push"]

[host]

It use https. As "# File generated by k3s. DO NOT EDIT.", I did not edit this file. I edit /etc/rancher/k3s/registries.yaml.
from

mirrors:
  "*":
    endpoint:
      - "http://10.111.160.21:39435"
configs:
  10.111.160.21:39435:
    auth:
      username: admin
      password: Zbwt_123456

to

mirrors:
  10.111.160.21:39435:
    endpoint:
      - "http://10.111.160.21:39435"
configs:
  10.111.160.21:39435:
    auth:
      username: admin
      password: Zbwt_123456

Then /var/lib/rancher/k3s/agent/etc/containerd/certs.d/10.111.160.21:39435 become

# File generated by k3s. DO NOT EDIT.
server = "https://10.111.160.21:39435/v2"
capabilities = ["pull", "resolve", "push"]

[host]

[host."http://10.111.160.21:39435/v2"]
  capabilities = ["pull", "resolve"]

and pod can run(Although I'm not entirely sure why it works now. Even though there's still an HTTPS link in /var/lib/rancher/k3s/agent/etc/containerd/certs.d/10.111.160.21:39435/hosts.toml, it just works somehow. ).

Thank you very much for your help.

[brandond]

Yeah, I guess if I'd though about it I would have remembered that you'll need to use a specific entry for that repo in the mirrors section since you have a specific entry for it in the configs section - it's existence will prevent it from trying the wildcard since it does exist.

Why were you using the wildcard in the first place? You don't seem to be using your repo as a mirror for anything.