Remote covers in a single-user instance use a lot of space

[renatolond]

Hi! Thanks for the work in Bookwyrm!

I've been running bookwyrm for myself for quite a while (my instance has been live since around 2020) and by now there's a lot of covers (~7.0GB) and previews (~3.0GB).
I don't follow any users, so I'm not sure how all those are grabbed, but that's fine, I think I would like something like Mastodon's account prune or statuses remove, something that would remove books or users that I (well, users of the instance to be more precise) haven't interacted with.

Is that something that already exists? (FYI I'm currently on 0.7.5)

[prolibre]

This is related to a problem I raised: book covers are not deleted when you change a book's cover. And so they accumulate.
For my part, I wrote a little bash script that, once a week, deletes all files that are not linked to the database. It's a handmade solution until Bookwyrm takes care of it automatically. Perhaps we should also consider compressing the covers or imposing a size limit.

CF : #2996 && #3266

[renatolond]

@prolibre Do you have a gist with that script somewhere? I'd love to test it over here too :)

[prolibre]

@renatolond I'm sorry. These are ‘home-made’ scripts that are really not ready to be shared (since they would need to be included in Bookwyrm and I don't have the Python skills).
Basically, I make a list of ‘orphaned’ files by browsing the ‘/images/covers’ folder and checking, one by one, whether they are present in the “bookwyrm_book” table (in the ‘cover’ field).

Then I use this list to delete each file and all its thumbnails (inside ‘thumbnails/covers’ in a subfolder with the same name as the image).
I run this script once a week and delete dozens/hundreds of images each time.
The first time, I ran the operation gradually (in batches of 500, I think) because the first run involved thousands of files.

I hope these explanations help you and clear things up a bit :-)

[prolibre]

@renatolond

So your request prompted me to check/fix/comment (in French, sorry) my script. So I'm leaving it here for you.
Two things:

• It's written in PHP... I don't know Python, which is why I didn't even try to integrate it into Bookwyrm. It can therefore be launched from the command line or via crontab.
• The script is designed for a Bookwyrm installation without Docker. You just need to modify a few lines of code to adapt it to your own installation.
  There you go... hope it helps :-)

nettoyage-images.php

[renatolond]

Thanks a lot, I'll check it out!

[prolibre]

@renatolond
You're welcome! I'd be curious to know how many files would be affected now? At the time, I had already deleted 19,000 images (including thumbnails), but I imagine that number has increased.

[renatolond]

I converted the script to Python because I don't have PHP in the machine I run bookwyrm and got 90881 files that would be deleted without dry-run. I haven't tried running it removing yet, because I have to double check if my script is correct, but I think it makes sense!

[renatolond]

Ran it this morning and got it down from 7GB to 2.1Gb 🙌🏼
Also made a slightly modified version of it for the previews folder, that one cleared from 3GB to 2.1GB

Here are both files in case it helps someone else:

clean_previews.py
clean_images.py

Thanks a lot, @prolibre, your script helped wonders 🙏🏼