replication over unreliable Networks / network partitions / disconnects

[aldas]

How good/bad would litestream work in situation where there are unreliable network connections involved.

For example: we have a fleet of vessels that have cellular network near port (99+% of time) but there are occasions when those vessels are so far off at sea that there are no persistent connection. This can last from couple of hours (most of the time) to extreme a week.

I would guess it is same as network partition between 2 servers in usual data center setup.

[benbjohnson]

Hi @aldas. Good question. Litestream copies out WAL frames out to a directory next to your database file so it doesn't interfere with SQLite's checkpointing. That will keep growing until it can flush out to the replica. That's probably your biggest concern is that you could fill up your disk depending on how many writes you're doing while disconnected.

Litestream will keep retrying though so disconnects aren't a problem.

[aldas]

Thanks, for our case changes piling up is not a problem as in normal conditions there are very small number of changes in database for one day.

just out of curiosity what would be the proper solution for this case? For example for our case (databases are small - below 1mb) it would not be a problem if whole database file would be uploaded from certain point and if that succeeds then resume by uploading changes.

[benbjohnson]

You shouldn't need any special configuration for Litestream. I think it should just work.

You can also look at using the session extension if you have a bunch of edge databases and you want to merge all changes into a single database periodically.