ErrImagePull after k3d image import using latest tag

[tekumara]

What did you do

• How was the cluster created?

  • k3d cluster create demo -p "8081:80@loadbalancer" --wait

• What did you do afterwards?

$ k3d image import myapp:latest -c demo
INFO[0000] Importing image(s) into cluster 'demo'    
INFO[0000] Loading 1 image(s) from runtime into nodes... 
INFO[0007] Importing images '[myapp:latest]' into node 'k3d-demo-server-0'... 
INFO[0024] Successfully imported image(s)               
INFO[0024] Successfully imported 1 image(s) into 1 cluster(s) 

$ docker exec -it k3d-demo-server-0 crictl images
IMAGE                                      TAG                    IMAGE ID            SIZE
docker.io/library/myapp                    latest                 06f7f2a4f851c       587MB
docker.io/rancher/coredns-coredns          1.8.3                  3885a5b7f138c       12.9MB
docker.io/rancher/klipper-helm             v0.6.6-build20211022   194c895f8d63f       84.5MB
docker.io/rancher/klipper-lb               v0.3.4                 746788bcc27e2       3.33MB
docker.io/rancher/library-traefik          2.4.8                  deaf4b1027ed4       27.8MB
docker.io/rancher/local-path-provisioner   v0.0.19                148c192562719       13.6MB
docker.io/rancher/metrics-server           v0.3.6                 9dd718864ce61       10.5MB
docker.io/rancher/pause                    3.1                    da86e6ba6ca19       327kB

$ kubectl run -it --image=myapp:latest myapp
error: timed out waiting for the condition

$ kubectl get events --sort-by='{.lastTimestamp}'
LAST SEEN   TYPE      REASON                    OBJECT                       MESSAGE
7m44s       Normal    Pulling                   pod/myapp                    Pulling image "docker.io/library/myapp:latest"
7m38s       Warning   Failed                    pod/myapp                    Failed to pull image "docker.io/library/myapp:latest": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/myapp:latest": failed to resolve reference "docker.io/library/myapp:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
7m38s       Warning   Failed                    pod/myapp                    Error: ErrImagePull
7m26s       Warning   Failed                    pod/myapp                    Error: ImagePullBackOff
4m13s       Normal    BackOff                   pod/myapp                    Back-off pulling image "docker.io/library/myapp:latest"

What did you expect to happen

Image is found and pod started.

Which OS & Architecture

darwin/amd64

Which version of k3d

k3d version v5.2.2
k3s version v1.21.7-k3s1 (default)

Which version of docker

docker version

Client:
 Cloud integration: v1.0.22
 Version:           20.10.11
 API version:       1.41
 Go version:        go1.16.10
 Git commit:        dea9396
 Built:             Thu Nov 18 00:36:09 2021
 OS/Arch:           darwin/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.11
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.9
  Git commit:       847da18
  Built:            Thu Nov 18 00:35:39 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.12
  GitCommit:        7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc:
  Version:          1.0.2
  GitCommit:        v1.0.2-0-g52b36a2
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.7.1)
  compose: Docker Compose (Docker Inc., v2.2.1)
  scan: Docker Scan (Docker Inc., v0.14.0)

Server:
 Containers: 116
  Running: 3
  Paused: 0
  Stopped: 113
 Images: 51
 Server Version: 20.10.11
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7b11cfaabd73bb80907dd23182b9347b4245eb5d
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.10.76-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 7.773GiB
 Name: docker-desktop
 ID: N6FS:ZMSR:L6NJ:2AZY:TTDX:ULSI:RM3M:3YGV:T5FK:WO6H:M3TX:LUOU
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 62
  Goroutines: 59
  System Time: 2022-01-11T06:12:04.498921447Z
  EventsListeners: 3
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

[tekumara]

The above does not happen when using a tag other than latest, eg:

k3d image import myapp:local -c demo
kubectl run -it --image=myapp:local myapp

[iwilltry42]

Hi @tekumara , thanks for opening this issue!
This is actually pretty interesting and I can confirm your experience there.
Now this doesn't seem to be an issue on k3d's side though, as everything it does works exactly as expected 🤔

UPDATE: Actually, it seems like it's expected that imagePullPolicy is Always if latest tag is specified and you didn't manually set a different policy: see https://kubernetes.io/docs/concepts/containers/images/#imagepullpolicy-defaulting

[tekumara]

Thanks for investing @iwilltry42! The imagePullPolicy defaulting explanation makes sense.