FailoverHeartbeatTTL should be configurable

[tommyalatalo]

Proposal

The FailoverHeartbeatTTL used when the leader node goes down is hard coded to 5 minutes.
This is a very long time in our use case where we're running clients and servers colocated on 3 nodes.
We need this parameter to be configurable so that it can be lowered to make recovery on colocated server+client nodes faster, since the tasks from the lost client node will not be replaced until the FailoverHeartbeatTTL time expires.

Short: make FailoverHeartbeatTTL configurable in the server config file.

Use-cases

Faster recovery when running a cluster with co-located server and client nodes, for instance 3VMs with nomad server+client on each, and no other nodes in the cluster.

Attempted Solutions

None available, FailoverHeartBeatTTL is hard coded and not exposed as a configuration parameter.

[shoenig]

Hi @tommyalatalo thanks for the issue. The heartbeating mechanism is actually rather complex; that hard-coded 5 minute TTL is only the initial value used. Over time, Nomad re-calculates the heartbeat interval per-Client so that the Servers avoid getting overloaded with heartbeat RPC requests. There's three config values for controlling hearbeat behavior, hearbeat_grace, min_heartbeat_ttl, and max_heartbeats_per_second, however all of them are oriented around increasing the time before fail-over. If we made a change to support capping heartbeat timing I suspect we could add something like max_heartbeat_ttl, though there's quite a few edge cases to think about with the large number of timeouts throughout the system (all the way from RPC limits to task kill timeouts, etc.) - I suspect the large 5 minute default timeout paves over a lot of that detail.