docker now uses containerd, and /var/lib/docker has moved to userdata, so why hasn't /var/lib/containerd moved too?

[samarium]

/var/lib/docker has moved to /mnt/deitpi_userdata/docker_data, presumably to move userdata off root file system.
Since docker now uses containerd which also stores images in /var/lib/containerd which is also bulk userdata, it seems that /var/lib/containerd should be moved to /mnt/dietpi_userdata/containerd_data, and /etc/containerd/config.toml adjusted with root=/mnt/dietpi_userdata/containerd-data.

I guess that dietpi_software needs to be hacked appropriately too.

[time passes ...]

moving /var/lib/containerd and updating config.toml worked

Looking at dietpi-software it already does config injection on config.toml for a different reason, so no reason why it should not do the relocation at installation like it does the /var/lib/docker relocation.

Currently not sure what happens when I relocate dietpi_userdata and I have already moved /var/lib/containerd to userdata.

dietpi-drive_manager does a dietpi-services stop which stops docker, but not containerd, so needs to know the dependency if moving containerd to userdata. I'll need to manually stop dietpi-servers and then systemctl stop containerd. Would have been simpler if waited to relocated containerd until after I had relocated userdata.

Also seems like dietpi-software doesn't handle relocated userdata, as lots of hardcoded /mnt/dietpi-userdata paths.
... Or it does by symlinking the default location to the new location :-)

Checking sizes shows containerd has the bulk of the userdata for me, which I want out of root, but then I use bind mounts for the majority of my docker volumes for my data to allow easier backups.

root@DietPi:/# du -hs /mnt/dietpi_userdata/*-data
505M	/mnt/dietpi_userdata/containerd-data
236K	/mnt/dietpi_userdata/docker-data

Maybe dietpi-services needs to know the docker is dependent on containerd and they should be stop/started in sync. Could probably do something in systemd, but maybe too complicated, and just have dietpi-services know and handle the dependency would be better.

[MichaIng]

Docker now uses /var/lib/containerd by default for images? That would be new to me. While it uses containerd since a long time, it still used /var/lib/docker (or in our case /mnt/dietpi_userdata/docker-data) for images pulled with docker pull or docker run, if not manually configured otherwise. However, while /var/lib/containerd may contain the images (which can be re-downloaded any time), the Docker data still contains the volumes created with docker volume create foo. Having images and the default volume path separated would even better match our typical /mnt/dietpi_userdata usage.

/mnt/dietpi_userdata is primarily meant to contain userdata and configs one might want to backup separately, move to a different/new system, cannot restore easily etc, which does not apply to Docker images. What benefit do you expect from having container images in /mnt/dietpi_userdata resp. outside of the rootfs?

Hardcoded /mnt/dietpi-userdata paths are not an issue: When moving it to a different drive via dietpi-drive_manager, a symlink is left behind to keep /mnt/dietpi-userdata paths intact.

AFAIK dietpi-services not controlling containerd was an oversight at first. But IIRC running containers can stall in a bad state if containerd is stopped for longer. The Docker daemon itself is only used for managing containers, creating new ones, starting/stopping them etc, but is not needed by running containers. But containerd ... yeah, I did a quick search and containers stop functioning and even data may be lost when stopping containerd without stopping containers first, so that is not a good idea and the way we handle it right now is correct.

[samarium]

I think you should look at where things are stored after pulling a few images, as from what I can see the image data is managed by containerd. I'm not sure where things split out, but for my example above I have containerd data 500M+, docker data < 1M. IIRC I measured while services and containers were stopped. I agree that previously all the data was in docker data, and I have been isolating that from rootfs for years. This is my first exposure to containerd and I was surprised to find all the data locked up in the container images, and not much in docker data.

You don't see true numbers when containers are active. I guess overlayfs or bind mounts from containerd to docker. See below for the major chunks of data in each of docker and containerd in various states of activity.

FYI, I'm using dietpi trixie 9.20+ and installed docker using dietpi-software.

# docker stop dns-server;  docker stop portainer_agent
# systemctl stop docker.socket docker containerd
# du -hs /mnt/dietpi_userdata/*-data/*{overlayfs,rootfs}
364M	/mnt/dietpi_userdata/containerd-data/io.containerd.snapshotter.v1.overlayfs
0	/mnt/dietpi_userdata/docker-data/rootfs

# systemctl start containerd docker docker.socket
# du -hs /mnt/dietpi_userdata/*-data/*{overlayfs,rootfs}
364M	/mnt/dietpi_userdata/containerd-data/io.containerd.snapshotter.v1.overlayfs
0	/mnt/dietpi_userdata/docker-data/rootfs

# docker start portainer_agent; docker start dns-server
# du -hs /mnt/dietpi_userdata/*-data/*{overlayfs,rootfs}
364M	/mnt/dietpi_userdata/containerd-data/io.containerd.snapshotter.v1.overlayfs
355M	/mnt/dietpi_userdata/docker-data/rootfs

Again if you are moving docker-data off rootfs to isolate ot from rootfs and to make it easier to backup seperately, then conainterd-data seems to be integral, and you can't meaningfully backup one without the other. I can snapshot a btrfs or lvm userdata and then make a stable backup even while containers are running, no worse that a system crash/reboot. I would prefer to avoid rootfs as I usually minimise the backups of that as it a relatively easily rebuilt with your dietpi automated install and a little other config data like from /etc and my normal ansible system setup scripts.

I have a small rootfs, and try to seperate most userdata from the system, and to stop userdata potentially filling rootfs, so it is on another disk, just like I would do with kvm/qemu images and other large data hog apps that aren't really part of the base system, and I like how you have seperated things like docker, syncthing, samba etc into userdata.

[MichaIng]

I checked it a bit, and volumes are still stored at the Docker data dir below /mnt/dietpi_userdata/docker-data/volumes/. Images are indeed stored by the containerd snapshotter, and then mounted (not actually stored) below /mnt/dietpi_userdata/docker-data/rootfs while the container is running.

Usually, the images can be restored as easily as a docker pull or docker run. Variable data is (supposed to be) stored in volumes or bind mounts, which can be moved over to new container images. Otherwise, when updating a container image, which means deleting the old image, would remove all variable data.

So what I said above, as far as I can see, remains true: Moving containerd data to /mnt/dietpi_userdata would be for reducing rootfs disk usage, in case performance (if the external disk is faster than rootfs), but would not help for easier backups/moving/restoring data etc.

[samarium]

The picture you are painting is that containerd is largely a cache, and wiping and recreating by docker compose/run would be sufficient, and would also signidicantly reduce my backup requirements. I'd be effectively moving /var/lib/containerd to /var/cache/containerd and not bother backing it up like I don't with most other things in /var/cache and accepting the various small issues resulting.

I've been bitten in the past with an inconsistent backup of /var/lib/docker, so I don't want to go there again. I don't have much use for docker volumes except for temporary backing store for non r/o container images, normal r/w volume I prefer to bind mount so I can backup/restore each application's data / config in one place simply.

Reducing rootfs is good for me, since I like to run rootfs around 4GB, and containerd was chewing up lots of that, but I'm happy enough to do it manually now I understand you position better, and I can add a few lines to my post install script if I requre automation, or write some ansible.

Thanks again for looking into and clarifying containerd in dietpi.

[MichaIng]

The picture you are painting is that containerd is largely a cache

No this is not what I wanted to say. A cache would be files which could be removed and the software would just re-create them automatically. /var/lib/containerd however contains the actual container image layers. So this can be better compared to libraries and executables below /lib /bin etc, which are essential for the software to work. But same as with Docker images, those files are no user data or configs or any such, but they can be easily recreated with a package (re)install.

and wiping and recreating by docker compose/run would be sufficient, and would also signidicantly reduce my backup requirements

Yes, it shouldn't be needed to include Docker images (/var/lib/containerd) into a user data backup, unless you want to do a full system backup. I did never test it, but I would assume that volumes below /mnt/dietpi_userdata/docker_data and images below /var/lib/containerd can be recovered mostly separately, or more importantly that the volume remains intact and can be reused even if the image/container is broken. But you might need to remove and re-create the old/broken container+image and pull a fresh one which re-uses the volume.

Would actually be a reasonable test:

1. Install Portainer
2. Change some Portainer settings
3. Copy /mnt/dietpi_userdata/docker_data to another fresh system
4. Install Portainer there
5. See whether Docker data including the volume from /mnt/dietpi_userdata/docker_data have been recovered as intended

[samarium]

I understand your point on containerd. While it may not actually be a cache, for my purposes it may as well be if my recovery method is reinstall and pull as required. I might try using ctr to delete an image corresponding to a non-running docker container, and then a running docker container, and a few other variations, and see what breaks. I have compose files, docker run scripts, and docker network configuration scripts for anything required to rebuild docker configuration anyway, so backups of docker configuration is probably gilding the lily, but would make recovery quicker and less complex, which is a good enough reason.

As for portainer, I already have a RUN script and bind mount portainer's /data so I can easily backup the data without all the baggage of docker volumes and the rest of the docker configuration, especially when I want to do a selective restore. For a selective restore I will usually just rollback a snapshot however as that is much easier even if I have more initial setup to have separate snapshot capable filesystems for each app. I need to investigate btrfs snapshots more for the dietpi side, currently using rsnapshot. Main server where portainer runs is using zfs snapshots which get replicated to the backup server, with an rsnapshot paranoia layer.

Except for initial testing purposes, I almost always setup the compose file for bind mounting app data, which works fine for most of my simple setups. I guess that non-RO rootfs images might exist temporarily when running, maybe the app modifies it normally, or maybe I have an entrypoint/startup script that hacks it to my requirements, but that will happen again automatically when a new container is instantiated anyway, so not something I care about backing up. Maybe swarm, kubernetes etc might require me to revisit my assumptions, however I don't need or want that complexity yet.

My hope is that my docker_data is almost only docker configuration, and of low volatility, and no docker volume that I care about.

I will probably look at doing a recovery test in a while, once I do some more hacking of the dietpi autoinstall script to hack the dietpi install/software scripts to make apt sources http again so they go thru my apt-cacher-ng proxy as I iterate thru testing dietpi autoinstall rather than going offsite every iteration. Sources which are https aren't happy using apt-cacher-ng and I haven't figured out how to do MITM with my CA cert as yet.