KConnect like functionality

[ssteidl]

I've worked on various embedded linux projects where we always use a bus with zmq based publish/subscribe for IPC. Obligatory it would be cool if...

There was a way to publish changes read from the wal. KConnect does this by reading the postgres wal file and publishing changes to kafka. This would be a great ipc system with durable and potentially replayable functionality.

Anyway just a thought on a fun use case. I'd use it :)

[benbjohnson]

Thanks for the suggestion. I think it's technically possible to take a given DB page in the WAL and traverse up to the root using the PTRMAP and from there determine the table it is in. That would allow Litestream to know if an INSERT, UPDATE, or DELETE occurred and for which record. Then it could execute a callback for each logical change to the database. Is that what you're thinking?

I've thought about doing something like that because I want to be able to have read replicas that may have caching enabled and I want to know when to invalidate the cache. However, it's definitely pretty complex so it'll take some testing to get right.