Master and Satellite report: "Zone 'master' is not connected. Log lag: 1 second" after deployment from Director

[peter-snr]

Describe the bug

After making changes in Icinga Director and executing a deployment we see incidents of 'Zone is not connected, these clear after about 3-5 mins.

To Reproduce

1. Make a change to a host in Director
2. Deploy the changes

Expected behavior

No alerts about Masters or Satellites not being connected.

Screenshots

Your Environment

version: 2.12.2-1
System information:
Platform: CentOS Linux
Platform version: 7 (Core)
Kernel: Linux
Kernel version: 3.10.0-1127.13.1.el7.x86_64
Architecture: x86_64

Build information:
Compiler: GNU 4.8.5
Build host: runner-hh8q3bz2-project-322-concurrent-0
OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017

Enabled features:
api checker ido-mysql mainlog notification

Icinga Web 2 Version
2.8.2
Git commit 8a89839af94a247ee2149b2336c73b8251b477c0
PHP Version 7.3.20
Git commit date 2020-08-17

Additional context

Add any other context about the problem here.

[0xliam]

This is also a bit of a pain point for us - we use the zone check on all of our satellites and agents, and we often see our critical services jump from 0 to 100 for a few minutes while the agents are reloading the new config.

[dgoetz]

How does your zone setup look like and in which direct are connections established?

I have large environments at customers which are fine with the cluster_zone check showing sometime an unknown after reload but no longer than 1 minute which is still a soft state, even in multi-tier setups with 2 or 3 levels of satellites.

In all of this environments I sync the director-global zone only to masters and satellites, agents have their own zone linux-agents and windows-agents so they only reload when a check command needs to be synced. Connection is always bi-directional for master and satellites so they reconnect as fast as possible, agents connect to master or satellite zone so no need for the central systems to try to establish connects for unavailable agents which reduces load but of course increases time to reconnect too.

[peter-snr]

Two Masters HA, 6 Satellites, Top down. All monitored boxes have icinga2 agents set up with zones.conf as per Icinga Docs.

/*
  * This is the zone.conf for the Primary Master
  */
 // Primary master
object Endpoint "aabd-sdmon-mstr01.somedomain.net" {
  // This server
}
// Secondary master
object Endpoint "aabc-sdmon-mstr02.somedomain.net" {
  // Actively connect to the secondary master
  host = "aabc-sdmon-mstr02.somedomain.net"
}
// Satellite Endpoints
object Endpoint "aabd-emea01-sat01.somedomain.net" {
  // Actively connect to satellite
  host = "aabd-emea01-sat01.somedomain.net"
}
object Endpoint "aabd-emea02-sat01.somedomain.net" {
  // Actively connect to satellite
  host = "aabd-emea02-sat01.somedomain.net"
}
object Endpoint "aabd-core01-sat01.somedomain.net" {
  // Actively connect to satellite
  host = "aabd-core01-sat01.somedomain.net"
}
object Endpoint "aabd-sdmon-sat01.somedomain.net" {
  // Actively connect to satellite
  host = "aabd-sdmon-sat01.somedomain.net"
}
object Endpoint "aabc-sdmon-sat01.somedomain.net" {
  // Actively connect to satellite
  host = "aabc-sdmon-sat01.somedomain.net"
}
object Endpoint "aabb-coemea03-sat01.somedomain.net" {
  // Actively connect to satellite
  host = "aabb-coemea03-sat01.somedomain.net"
}
// MASTER ZONE
object Zone "master" {
  endpoints = [ "aabd-sdmon-mstr01.somedomain.net", "aabc-sdmon-mstr02.somedomain.net" ]
}
// EMEA01 Satellite
object Zone "aabd-emea01-satellite" {
  endpoints = [ "aabd-emea01-sat01.somedomain.net" ]
  parent = "master"
}
// EMEA02 Satellite
object Zone "aabd-emea02-satellite" {
  endpoints = [ "aabd-emea02-sat01.somedomain.net" ]
  parent = "master"
}
// CORE01 Satellite
object Zone "aabd-core01-satellite" {
  endpoints = [ "aabd-core01-sat01.somedomain.net" ]
  parent = "master"
}
// SDMON Satellite
object Zone "aabd-sdmon-satellite" {
  endpoints = [ "aabd-sdmon-sat01.somedomain.net" ]
  parent = "master"
}
// Satellite
object Zone "aabc-sdmon-satellite" {
  endpoints = [ "aabc-sdmon-sat01.somedomain.net" ]
  parent = "master"
}
// COEMEA03 Satellite
object Zone "aabb-coemea03-satellite" {
  endpoints = [ "aabb-coemea03-sat01.somedomain.net" ]
  parent = "master"
}
object Zone "global-templates" {
    global = true
}
object Zone "director-global" {
    global = true
}



 /*
  * Satellite zone.conf
  */
 object Endpoint "aabd-sdmon-mstr01.somedomain.net" {
         // This endpoint will connect to us
 }
 object Endpoint "aabc-sdmon-mstr02.somedomain.net" {
         // This endpoint will connect to us
 }
 object Endpoint "aabd-sdmon-sat01.somedomain.net" {
     // This server
 }
 object Zone "master" {
         endpoints = [ "aabd-sdmon-mstr01.somedomain.net", "aabc-sdmon-mstr02.somedomain.net" ]
 }
 object Zone "aabd-sdmon-satellite" {
         endpoints = [ "aabd-sdmon-sat01.somedomain.net" ]
         parent = "master"
 }
 /*
  * Defines a global zone for distributed setups with masters,
  * satellites and clients.
  * This is required to sync configuration commands,
  * templates, apply rules, etc. to satellite and clients.
  * All nodes require the same configuration and must
  * have `accept_config` enabled in the `api` feature.
  */
 object Zone "global-templates" {
   global = true
 }
 /*
  * Defines a global zone for the Icinga Director.
  * This is required to sync configuration commands,
  * templates, apply rules, etc. to satellite and clients.
  * All nodes require the same configuration and must
  * have `accept_config` enabled in the `api` feature.
  */
 object Zone "director-global" {
   global = true
 }

/*
 * Agent zone.conf
 */
// Define Endpoints
object Endpoint "aabd-sdmon-sat01.somedomain.net" {
}
object Endpoint "aabdsdbastion.somedomain.net" {
        log_duration = 0
}
// Define Zones
object Zone "aabd-sdmon-satellite" {
        endpoints = [ "aabd-sdmon-sat01.somedomain.net" ]
}
// Agent has its own zone.
object Zone "aabdsdbastion.somedomain.net" {
        endpoints = [ "aabdsdbastion.somedomain.net" ]
        parent = "aabd-sdmon-satellite"
}
object Zone "global-templates" {
        global = true
}
object Zone "director-global" {
        global = true
}

[Al2Klimov]

Hello @peter-snr and thank you for reporting!

Is any of the following workarounds reasonable for you?

• Increase max_check_attempts
• Schedule downtimes before deploying

Best,
AK

[peter-snr]

Hi @Al2Klimov

the workarounds, apply to masters and satellites only?

Cheers
PG

[Al2Klimov]

They apply to all zone check services.

[peter-snr]

Ah, makes sense, I'll give it a try.

[peter-snr]

@Al2Klimov - I increased max_check_attempts to 15 a week ago. But we are still seeing the same issue. It's not practical to put in downtime when deploying/updating commands, especially for multiple hosts.

Issue still exists.

[Al2Klimov]

Wait... these are soft states! So you don't get actually alarmed. Isn't this reasonable?