Disintegrate 4.0, Postgres Sequences and global event ordering.

[phillipbaird]

Just been looking at the changes in 4.0 and wondering if I could confirm my understanding how using a Sequence for the event id works?

Quite some time ago I read an article that claimed you cannot use Postgres sequences for event ids because the allocation order of the sequence is not guaranteed to be the same order as transaction commits. Serialization isolation does not help because as far as I understand, the sequences are non-transactional and are not considered. e.g. you could allocate event ids 101, 102, but the transactions for those events could commit in the opposite order, 102 then 101.

So assuming the above is valid I'm wondering how Disintegrate 4.0 guarantees correct event ordering?

Thanks.

[luca-iachini]

Hi @phillipbaird,
thanks for bringing up this topic, and apologies for the late reply.

This discussion helped us realize that, while the latest version introduced several improvements and made the code cleaner, we accidentally removed one of the most important concepts from v3 that was actually preventing the issue you correctly described, the "epoch" functions.

event_store_begin_epoch is needed to signal that concurrent and in-flight transactions are writing to the event store. Each concurrent append records the last event_id in the sequence at the moment it starts. During streams, we then verify that all retrieved event Ids are lower than both (event_store_current_epoch):

• the event Ids associated with in-flight transactions, and
• the current maximum event store ID.

This guarantees that no new event with an Id lower than the computed epoch can appear after the stream has ended.

To validate this behavior, we added two tests demonstrating that the epoch is computed correctly and that the stream cannot skip events:

disintegrate/disintegrate-postgres/src/event_store/tests.rs

Line 281 in c17ef14

async fn stream_does_not_skip_events_when_writers_commit_out_of_order(pool: PgPool) {

disintegrate/disintegrate-postgres/src/event_store/tests.rs

Line 409 in c17ef14

async fn epoch_does_not_exceed_max_committed_event_id(pool: PgPool) {

These tests will help us prevent this bug in future releases.

We have now released disintegrate-postgres 4.0.1, which fixes the issue.

Thanks again for raising this 🙏