Fault-tolerance configuration for messaging/rabbitmq

[igcherkaev]

@nmaludy asked to open an issue to discuss making certain retry parameters configurable.
The idea here is to make at least these two configurable via st2.conf:

https://github.com/StackStorm/st2/blob/v3.2/st2common/st2common/transport/connection_retry_wrapper.py#L33-L35

I see them in the [messaging] section: https://github.com/StackStorm/st2/blob/v3.2/conf/st2.conf.sample#L183-L203

And also make the following errors visible without enabling debug to allow users see/monitor errors affecting st2 performance:

1. https://github.com/StackStorm/st2/blob/v3.2/st2common/st2common/transport/connection_retry_wrapper.py#L143-L144
2. https://github.com/StackStorm/st2/blob/v3.2/st2common/st2common/transport/connection_retry_wrapper.py#L155-L156

And last, but not least, perhaps it'd be nice to implement rabbitmq heartbeat feature to ensure connections are kept alive if they are traversing firewalls or other connection trackers?

Per @nmaludy :

heartbeat (float) – Heartbeat interval in int/float seconds. Note that if heartbeats are enabled then the heartbeat_check() method must be called regularly, around once per second.

Which, I believe, is not being used within st2 source code.

[arm4b]

To cross-link related items, here is an issue for adding RabbitMQ heartbeat configuration in st2.conf: #4780