Improving fault-tolerance by adding a failover node #378
Description
As far as I could find, a single EthSigner instance always connects to exactly 1 Besu node.
When this Besu node is unavailable, the EthSigner instance will be unavailable as well, resulting in faults to cascade.
Ideally, we would like an EthSigner instance to connect to a different Besu node, if this node is offline.
For instance, we could allow multiple hosts and IPs to be configured in the downstream as a comma-separated list rather than a single value. Or, perhaps EthSigner could tap into the peer discovery to automatically populate a list of available nodes.
In any case, there is room for improvement on the fault-tolerance aspect. If we want high availability now without signing transactions locally, we need multiple Besu nodes and multiple EthSigner instance per node (in case EthSigner crashes but the node stays operational).
Ideally, if an EthSigner node is offline, we send a tx to another EthSigner node. Or, if a Besu node is offline, the EthSigner node should resolve this by sending the tx to a different node.