Stopping of isolated primary #53
Description
A nice feature would be to automatically bring down a primary server to which no or too few standby servers are connected. This would help to minimize the impact of a split brain. This could be implemented by calling a script specified in the configuration. Repmgr from 2ndQuadrant has something like this.
An example setup for which this would useful:
We have three sites. Site [1] with the primary database, site [2] with a standby, site [3] with another standby that is used as quorum/observer.
The primary [1] uses synchronous replication, preferably to standby [2] with the quorum/observer [3] as backup. After a failover, [2] will do the same.
If the primary [1] becomes isolated, pglookout will make the standby [2] (also) primary and make the quorum/observer [3] follow this new primary [2].
The old primary [1] will be unable to process write transactions due to synchronous replication and no connected standby,
Applications use a JDBC-connector with both [1] and [2] in it and targetServerType=master, to connect only to primary servers. In this case, though, there are two, and the application may also connect to the old primary [1]. Write transactions will hang, but this is still not optimal.
So it would help if pglookout would see that there are no standby servers connected anymore to the old primary [1] and have it start a script that would bring it down.