Separate archives per volume (feature request or question)

[unleashit]

Hi, thanks for your work. I think this is a feature request unless I haven't figured out how to accomplish what I'm trying to do.

Currently, I have a single docker-volume backup container backing up volumes from many other docker compose files via a shared network. Right now, each volume mounts to its own subfolder under /backup. Since I'm using Kopia to manage database and other backups, I've set the cron on DVB to never and just use Kopia pre/post scripts to start and cleanup the DVB backup. This is working pretty well, except I end up with one very large archive of all the backed up volumes. That's somewhat of a problem because it goes to Backblaze b2, where there is a maximum daily download size (for the free account)... which means at a certain size, at least I assume, it may become impossible to download.

Question: is there currently a way to output multiple backup archives within the same DVB instance? If not I think it would be a great addition :)

[m90]

I don't think this is currently possible, but I'm also not sure I really understand your setup.

I've set the cron on DVB to never

Then what do you use docker-volume-backup for if it's never run? Do you trigger the command manually?

is there currently a way to output multiple backup archives within the same DVB instance?

How would you imagine configuration for such a feature to work? I'm currently having a hard time coming up with an intuitive way of splitting the content other than resorting to multiple instances, each one of them responsible for a part of the data.

[unleashit]

Thanks for the reply. Basically, I've assigned scripts that run before/after Kopia snapshots, which run on their own schedule. In the pre script, I manually run the "backup" command inside the docker volume backup container. That spits out the volume backups as a single .tar into my local Kopia snapshot directory. Kopia then runs, uploads it to remote storage then runs the after script which deletes the local copy.

I still find DVB useful for this case, because it does a few things that would take time to manually wire up like stopping labeled containers first (and is compatible with Swarm). In the future, I may end up using other features as it's a handy tool.

How would you imagine configuration for such a feature to work?

The simplest thing that comes to mind would be to add an env var flag that when true, saves separate archives per subdirectory of /backup into /archive.

Another feature request relates to this as well I think, as they are also using a multiple compose file workflow. In my case, If I ran DVB containers for "each", I would have about 15 extra backup containers.

[unleashit]

After some more testing, it seems that the containers with the docker-volume-backup.stop-during-backup=true label that are outside of the docker compose file that docker volume backup is in aren't actually stopping during the backups. I didn't see that covered in the docs, but I saw a couple others in issues that seemed to be doing the same as me (DVM in one compose file, with an external network linking services in other compose files) so I assumed it could work. If that's not possible, I'll need another solution for the setup.

Then again, maybe its a swarm issue. I did try both methods on the docker swarm page and neither worked. Or at least running docker ps in a separate terminal window during the backup, showed them all running...

[m90]

I think this is basically #425. Is the node you are running the container on a manager node?

[unleashit]

On this issue, it was operator error! I'm new to docker swarm and only using it because Dokploy uses it. Dokploy is running on a single manager node, but the command it was issuing to deploy my compose projects is docker compose and not docker stack. It allows the customization of the command, and I may play with that in the future. But for now I fixed the problem of the containers not stopping by moving the labels to the service parent. I can now see it stopping the right containers in the DVB stdout.

[unleashit]

On the feature request, if you don't see it as a common use case I can implement it manually in my scripts. But even though that's possible, for performance reasons (handling the archiving process only once) it could be a handy feature.

[m90]

For me it's not so much about how common the use case is, it's also about a. is there a sensible interface to the feature and b. is it possible to implement it without having to change too much code.

I do think the feature you propose makes sense, but I'd have to think about it more, also in conjunction with #446. If you have any concrete ideas, feel free to note them down here or even try to implement them.

[unleashit]

What do you think of the idea of just adding an environment variable, where if true you do something like:

    find /backup -maxdepth 1 -type d \( ! -name . \) -print0 | while IFS= read -r -d '' dir; do
        dir_name=$(basename "$dir")
        tar -czf "/archive/${dir_name}.tar.gz" "$dir"
    done

Warning: _tty llm produced code warning just for demo purposes.

This would be basic and leave the user responsible to ensure they are using the right folder structure. Maybe a warning could be added if any loose files were found in the parent, or maybe not as long as the docs explained.

I'm not sure how this has to do with #425 ? Sorry for sort of hijacking my own thread with that other issue :) In the future I would be happy to try and take a look at that, but right now I'm not going to move my containers to Swarm for the time being. I'm also not experienced with Swarm yet.

[m90]

Trouble is this is more than just a shell script. Right now, there's the concept of a single file artifact which would have to be changed into a list of files, which will then need to be supported and changed for all storage backends.

I think it makes sense, but it's not trivial, so I am not able to make any predictions about how and when this will happen.