Add warning about systemd version for timeouts

[rjernst]

The timeout extension mechanism was added in systemd version 238. However, some Elasticsearch supported OSes have systemd versions older than this. sd_notify happily accepts any environment variables; it only fails if the underlying socket cannot be written to. This means on those older systems, we send timeout extensions into a black hole, they are silently ignored.

We should proactively detect the systemd version and warn that it is too old and the timeouts will not work, so the user should extend the service timeout themselves if necessary.

[elasticmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)

[philippkahr]

Running systemctl --version on a centos7 reveals 219 as a version.

@morashwan did an upgrade from a healthy three node cluster 7.17.5 to 8.3.2 and discovered that after 3 minutes the nodes just disappear / shutdown. When we increase the systemd timeout to three hours, this will happen in three hours.

My guess is that this This means on those older systems, we send timeout extensions into a black hole, they are silently ignored. is the case here.

According to the support matrix we should support CentOS 7 https://www.elastic.co/support/matrix

[rjernst]

According to the support matrix we should support CentOS 7

We can only support it to the extent the system allows us to support it. In this case, systemd is ignoring the newer mechanism we are using to notify it that the system is taking more time to startup than initially expected (75 seconds).

When we increase the systemd timeout to three hours, this will happen in three hours.

That sounds like something else may be going on; the issue here is when recoveries could cause startup to take longer. Thought perhaps recoveries are taking 3+ hours for you, but that is outside the scope of this issue. This issue is about adding a warning so that users know their systemd version is older than that which supports the timeout extension, so they will need to increase their timeout explicitly.