What happened?
I deployed a k8s cluster using CRIO as container runtime:
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
localhost Ready control-plane 6m44s v1.35.4 10.185.105.237 <none> Ubuntu 24.04.3 LTS 6.8.0-124-generic cri-o://1.35.0
ArgoCD failed to deploy:
NAMESPACE NAME READY STATUS RESTARTS AGE
argocd argocd-application-controller-0 1/1 Running 0 5m23s
argocd argocd-applicationset-controller-5cc599989f-2zk77 1/1 Running 0 5m24s
argocd argocd-dex-server-6bd7bd9c68-kg5tt 1/1 Running 0 5m24s
argocd argocd-notifications-controller-658879594b-cn92z 1/1 Running 0 5m24s
argocd argocd-redis-6d96789db9-rb5d5 0/1 ImagePullBackOff 0 5m23s
argocd argocd-repo-server-579b78689-tc55b 1/1 Running 0 5m23s
argocd argocd-server-574588fc44-hm84f 1/1 Running 0 5m23s
kube-system calico-kube-controllers-7c99df9fc4-s8l69 1/1 Running 0 6m24s
kube-system calico-node-rr2c2 1/1 Running 0 6m34s
kube-system coredns-58cc5d8ddf-xr9vn 1/1 Running 0 6m11s
kube-system dns-autoscaler-7ccf68df7c-s8nh5 1/1 Running 0 6m9s
kube-system kube-apiserver-localhost 1/1 Running 0 7m16s
kube-system kube-controller-manager-localhost 1/1 Running 0 7m16s
kube-system kube-proxy-wjvwf 1/1 Running 0 6m56s
kube-system kube-scheduler-localhost 1/1 Running 0 7m16s
kube-system nodelocaldns-8jml6 1/1 Running 0 6m5s
$ kubectl describe pod -n argocd-redis-6d96789db9-rb5d5
<SNIP>
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m55s default-scheduler Successfully assigned argocd/argocd-redis-6d96789db9-rb5d5 to localhost
Normal Pulling 6m55s kubelet spec.initContainers{secret-init}: Pulling image "quay.io/argoproj/argocd:v2.14.5"
Normal Pulled 6m51s kubelet spec.initContainers{secret-init}: Successfully pulled image "quay.io/argoproj/argocd:v2.14.5" in 1.035s (4.117s including waiting). Image size: 502604298 bytes.
Normal Created 6m50s kubelet spec.initContainers{secret-init}: Container created
Normal Started 6m50s kubelet spec.initContainers{secret-init}: Container started
Normal Pulling 3m52s (x5 over 6m49s) kubelet spec.containers{redis}: Pulling image "redis:7.0.15-alpine"
Warning Failed 3m52s (x5 over 6m47s) kubelet spec.containers{redis}: Failed to pull image "redis:7.0.15-alpine": short-name "redis:7.0.15-alpine" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf.d/01-unqualified.conf"
Warning Failed 3m52s (x5 over 6m47s) kubelet spec.containers{redis}: Error: ErrImagePull
Normal BackOff 107s (x20 over 6m46s) kubelet spec.containers{redis}: Back-off pulling image "redis:7.0.15-alpine"
Warning Failed 94s (x21 over 6m46s) kubelet spec.containers{redis}: Error: ImagePullBackOff
nvidia@localhost:~$ kubectl describe pod -n argocd argocd-redis-6d96789db9-rb5d5
What did you expect to happen?
Expectation is that ArgoCD deploys successfully
How can we reproduce it (as minimally and precisely as possible)?
Deploy K8s cluster with the following customization:
inventory/local/group_vars/k8s_cluster/k8s-cluster.yml
## Container runtime
## docker for docker, crio for cri-o and containerd for containerd.
## Default: containerd
container_manager: crio
inventory/local/group_vars/k8s_cluster/addons.yml:
argocd_enabled: true
argocd_namespace: argocd
OS
Ubuntu 24
Version of Ansible
$ ansible --version
ansible [core 2.18.17]
config file = None
configured module search path = ['/home/nvidia/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /home/nvidia/.local/lib/python3.12/site-packages/ansible
ansible collection location = /home/nvidia/.ansible/collections:/usr/share/ansible/collections
executable location = /home/nvidia/.local/bin/ansible
python version = 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (/usr/bin/python3)
jinja version = 3.1.2
libyaml = True
Version of Python
3.12.3
Version of Kubespray (commit)
2.31.0
Network plugin used
calico
Full inventory with variables
$ cat inventory/local/hosts.ini
localhost ansible_connection=local local_release_dir={{ansible_env.HOME}}/releases
[kube_control_plane]
localhost
[etcd]
localhost
[kube_node]
localhost
Command used to invoke ansible
ansible-playbook -i inventory/local/hosts.ini cluster.yml -b --ask-become-pass
Output of ansible run
TASKS RECAP ***********************************************************************************************************************************************************
Wednesday 10 June 2026 04:18:07 +0000 (0:00:00.095) 0:05:52.969 ********
===============================================================================
kubernetes/control-plane : Kubeadm | Initialize first control plane node (1st try) ---------------------------------------------------------------------------- 11.80s
container-engine/cri-o : Extract_file | Unpacking archive ------------------------------------------------------------------------------------------------------ 6.62s
kubernetes-apps/ansible : Kubernetes Apps | CoreDNS ------------------------------------------------------------------------------------------------------------ 6.57s
kubernetes-apps/helm : Download_file | Download item ----------------------------------------------------------------------------------------------------------- 5.69s
etcd : Configure | Check if etcd cluster is healthy ------------------------------------------------------------------------------------------------------------ 5.38s
kubernetes-apps/helm : Extract_file | Unpacking archive -------------------------------------------------------------------------------------------------------- 5.09s
etcd : Restart etcd -------------------------------------------------------------------------------------------------------------------------------------------- 5.04s
network_plugin/cni : CNI | Copy cni plugins -------------------------------------------------------------------------------------------------------------------- 4.65s
etcd : Configure | Ensure etcd is running ---------------------------------------------------------------------------------------------------------------------- 4.56s
etcdctl_etcdutl : Download_file | Download item ---------------------------------------------------------------------------------------------------------------- 4.07s
kubernetes-apps/external_provisioner/local_path_provisioner : Local Path Provisioner | Create manifests -------------------------------------------------------- 3.87s
kubernetes-apps/argocd : Download_file | Download item --------------------------------------------------------------------------------------------------------- 3.76s
kubernetes-apps/argocd : Kubernetes Apps | Install ArgoCD ------------------------------------------------------------------------------------------------------ 3.67s
download : Extract_file | Unpacking archive -------------------------------------------------------------------------------------------------------------------- 3.64s
etcdctl_etcdutl : Extract_file | Unpacking archive ------------------------------------------------------------------------------------------------------------- 3.50s
network_plugin/calico : Calico | Create calico manifests ------------------------------------------------------------------------------------------------------- 3.49s
container-engine/validate-container-engine : Populate service facts -------------------------------------------------------------------------------------------- 3.43s
kubernetes-apps/external_provisioner/local_path_provisioner : Local Path Provisioner | Apply manifests --------------------------------------------------------- 3.36s
container-engine/cri-o : Download_file | Download item --------------------------------------------------------------------------------------------------------- 3.31s
container-engine/crictl : Extract_file | Unpacking archive ----------------------------------------------------------------------------------------------------- 3.15s
Anything else we need to know
I asked CODEX about the fix and it answered with:
Fix this inventory file:
kubespray/inventory/local/group_vars/all/cri-o.yml
It currently only has commented examples. Add crio_registries there, for example:
crio_registries:
- prefix: docker.io
location: registry-1.docker.io
insecure: false
blocked: false
unqualified: true
Why this file: Kubespray uses roles/container-engine/cri-o/templates/unqualified.conf.j2 to render:
/etc/containers/registries.conf.d/01-unqualified.conf
Right now that rendered file contains:
unqualified-search-registries = []
That is exactly why CRI-O refuses redis:7.0.15-alpine.
After editing, rerun Kubespray against the active inventory:
cd kubespray
ansible-playbook -i inventory/local/hosts.ini cluster.yml -b
Then restart/delete the stuck Redis pod so it pulls again.
What happened?
I deployed a k8s cluster using CRIO as container runtime:
ArgoCD failed to deploy:
What did you expect to happen?
Expectation is that ArgoCD deploys successfully
How can we reproduce it (as minimally and precisely as possible)?
Deploy K8s cluster with the following customization:
inventory/local/group_vars/k8s_cluster/k8s-cluster.yml
inventory/local/group_vars/k8s_cluster/addons.yml:
OS
Ubuntu 24
Version of Ansible
Version of Python
3.12.3
Version of Kubespray (commit)
2.31.0
Network plugin used
calico
Full inventory with variables
Command used to invoke ansible
ansible-playbook -i inventory/local/hosts.ini cluster.yml -b --ask-become-pass
Output of ansible run
Anything else we need to know
I asked CODEX about the fix and it answered with: