How to prune snapshots with containerd?

[toxsick]

Since the switch to containerd the containerd snapshot folder grows on my host:

/var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/...

Now there there are no inodes and space left on the host. With docker I used to regulary prune old stuff. But how can I cleanup this artifacts with containerd / ctr?

Thanks in advance!

Hannes

[ktsakalozos]

Hi @toxsick I do not have a good answer (for now) but here is what i know. We can see the running containers with:

microk8s.ctr -n k8s.io containers ls

We can then remove images with:

microk8s.ctr -n k8s.io images rm --sync "image_ref"

Also kubelet should be garbage collecting unused images: https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/

[toxsick]

Hi @ktsakalozos

thanks for your answer! I saw that, but there does not seam to be something easy right? I would like to lower the image-gc-high-threshold to something like 60%. Any idea where I can configure this in microk8s/kubelet?

[ktsakalozos]

Any idea where I can configure this in microk8s/kubelet?

You can edit /var/snap/microk8s/current/args/kubelet to add the arguments you please and then do a microk8s.stop, microk8s.start cycle.

[toxsick]

Ohh that helps! Thanks!

[koss822]

Basically stop microk8s
microk8s.ctr containers ls | cut -f1 -d' ' | tail -n +2 | xargs -n1 -r microk8s.ctr containers delete remove all containers
microk8s.ctr images ls | cut -f1 -d' ' | tail -n +2 | xargs -n1 -r microk8s.ctr images rm remove all images
mv io.containerd.snapshotter.v1.overlayfs io.containerd.snapshotter.v1.overlayfs.old delete directory and recreate it
mkdir -p /var/snap/microk8s/common/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots

I wrote some tutorial over it here - https://www.martinkonicek.eu/posts/microk8s-containerd-cleanup/