HA: command endpoint checks are scheduled on a node even if it lost the connection to the agent

[julianbrost]

When you have a HA zone (sat-1, sat-2) that executes checks on child agents using the command endpoint feature, the checkables are distributed between the two HA nodes ignoring the current state of connections to the agents. So if only only one of the nodes is connected to the agent (agent), this check can still be executed on the other one, giving a Remote Icinga instance 'agent' is not connected to 'sat-1' result even though sat-2 would be able to execute the check.

[julianbrost]

This happens due to objects being distributed between endpoints in a HA zone only considering the set of endpoints using a hash function and only considering the set of currently online endpoints within this zone:

icinga2/lib/remote/apilistener-authority.cpp

Line 71 in aa4f530

authority = endpoints[Utility::SDBM(object->GetName()) % endpoints.size()] == my_endpoint;

So fixing this would either require significant changes to how objects are distributed, or trying to proxy the command endpoint through the other HA node if the current has no connection.

[Al2Klimov]

Well... all nodes of immediately related zones have to be connected to each other, scenarios as #6766 (comment) just don't work. I wouldn't relax that requirement.

[julianbrost]

I'm aware that this isn't a simple fix with the current design but more like something for a cluster communication redesign. Still I think this is something that should actually work. If a node is reachable from the parent zone at all, it should be possible to check it.

[julianbrost]

#9192 fundamentally described the same issue but brought up another detail: for cluster connectivity checks, this should also be handled in some way, like have "OK: redundant connection or non-HA setup", "WARNING: lost redundancy", "CRITICAL: lost connection" results.