PG to PG: Add ORDER BY in MERGE to handle non-PK unique columns

[Amogh-Bharadwaj]

Problem statement

Consider a Postgres table such with id PRIMARY KEY and email TEXT with a UNIQUE constraint on it. Let's say there is a mirror of this table to another PG database, and the following changes are made on source:

DELETE FROM table WHERE id = 1 (where email = 'alice@example.com')
INSERT INTO table (id, email) VALUES (2, 'alice@example.com')

This can result in a unique constraint violation error in normalize:

error executing normalize statement for table public.accounts: 
ERROR: duplicate key value violates unique constraint

This is because MERGE in Postgres processes rows in non-deterministic fashion, therefore across PKeys we could attempt replicating the INSERT first before DELETE (note: for rows pertaining to the same PKey this is not possible as we rank there), resulting in us hitting a non-PK unique constraint.

Fix

According to Postgres docs for MERGE, it says:

The order in which rows are generated from the data source is indeterminate by default. 
A source_query can be used to specify a consistent ordering, if required, which might be needed to avoid deadlocks between concurrent transactions.

We are already using the source_query syntax in our MERGE. This PR adds an ORDER BY to our subquery which sorts by _peerdb_timestamp which is representative of WAL order, thereby not running into this issue.

• Functionally tested

[heavycrystal]

Have we measured perf impact for large batch sizes?

[Amogh-Bharadwaj]

Have we measured perf impact for large batch sizes?

For large batch sizes the bottleneck will be the CTE taking too much memory or the MERGE I/O. Even for 10 million unique pkeys in a batch the sort itself shouldn't take more than a second

Definitely worth the R&D there though