Need to monitor latencies between nodes

[alexpyattaev]

Problem

We want to use existing ping mechanism in gossip to collect data about true latencies between validators.

• we only care about staked validators, and we care very much about >67% validators
• the distribution of validators may change, so we need to have semi-persistent monitoring going on
• all of this probably needs to be handled via metrics and be non-disruptive to the cluster.

Proposed Solution