Replication to untrusted clients

[kg]

Is there a way for me to use litestream to replicate subsets of my data to untrusted clients? The model I'm considering here is to slice the parts of my tables that relate to the user out of the server's database and ship them over so the client can update its local copy of the data (also a SQLite db). I've looked at some of the SQLite built-in options for generating and applying SQL diffs, but the performance characteristics seem questionable. It seems like if I could tap into the journaling/replication mechanisms litestream already has to generate per-user logs or something like that to ship over the wire to clients, it would solve my problem perfectly. If there's some way I could cobble that together it would be great to know (are the streams/logs litestream uses in a documented format I could just parse myself?)

[benbjohnson]

hey @kg, Litestream doesn't have a way to selectively replicate data unfortunately. It copies out whole pages and if you tried skipping pages then SQLite would probably see the database as corrupt (at least through a PRAGMA integrity_check point-of-view).

I've looked at some of the SQLite built-in options for generating and applying SQL diffs, but the performance characteristics seem questionable

Are you talking about the session extension? I haven't used it personally. What are the performance characteristics that seem questionable?

[kg]

I actually hadn't looked at the session extension, just some tools for diffing sqlite databases/tables that I saw in the documentation. That extension looks like it could be viable if I figure out how to use it. It looks like it doesn't store the session data into the database, so it shouldn't interfere with litestream.

[benbjohnson]

AFAIK the session extension just produces some binary data you can ship around. I’d be curious to hear how it goes if you get a chance to use it!