docker-image-cleaner using 100% of one CPU core

[rgaiacs]

@arnim noticed that docker-image-cleaner as configured at https://github.com/jupyterhub/binderhub/blob/3b3743b65a7cc7016a2823a4ea1ca3f1aacd03f3/helm-chart/binderhub/templates/image-cleaner.yaml#L8 is using 100% of one CPU core all the time.

@manics is this the expected behaviour?

[manics]

It's not. How much free disk space do you have?

[rgaiacs]

df -h returns

Filesystem                            Size  Used Avail Use% Mounted on
tmpfs                                  76G   11M   76G   1% /run
/dev/mapper/ubuntu--vg-ubuntu--lv      98G   36G   58G  39% /
tmpfs                                 378G     0  378G   0% /dev/shm
tmpfs                                 5.0M     0  5.0M   0% /run/lock
/dev/sda2                             2.0G  253M  1.6G  14% /boot
/dev/sda1                             1.1G  6.1M  1.1G   1% /boot/efi
/dev/sdd1                             6.0T  4.5T  1.2T  79% /orc2_data/containerd
/dev/mapper/mpatha-part1              6.0T  2.1T  4.0T  34% /orc_data/persistent_volume
/dev/mapper/spko--css--app03--vg-ssd  1.8T  1.2T  525G  69% /orc2_data/repo2docker

The disk used by JupyterHub is

Filesystem                            Size  Used Avail Use% Mounted on
/dev/sdd1                             6.0T  4.5T  1.2T  79% /orc2_data/containerd

[manics]

Can you try killing the pod, and see if it uses 100% CPU when it restarts?

[rgaiacs]

Restart the pod had no effect.

[manics]

Is there anything useful in the docker-image-cleaner logs? The actual code is relatively simple
https://github.com/jupyterhub/docker-image-cleaner/blob/main/docker_image_cleaner/cleaner.py

[rgaiacs]

Is there anything useful in the docker-image-cleaner logs?

No. I will look at this another time. #3160 takes priority.

2025-01-13 14:59:08,339 Pruning docker images when /var/lib/dind has 1490GB used
2025-01-13 15:17:56,866 1476.37GB used