Allow disabling Scheduling Offset for certain Checks

[oxzi]

When Icinga 2 schedules the next execution time of a check, the scheduling offset is taken into account. As documented there in prose and also discussed in #10255, the next scheduled execution time will be smaller or equal then current_time + check_interval. Or, in simpler words, the next execution will be too early, which was the reason for #10255.

When combining a check_interval and a check_period, this can result in unexpected behavior as shown in a community post.

Since I only found a documentation of this behavior in the linked "Scheduling offset" section of the "Technical concepts" chapter, but not in the earlier chapters, having such an offset is a footgun for users. If they would configure a check_interval of, for example, one minute, they would expect a minute and not 50-something seconds.

Thus, I would suggest to both document the scheduling offset more prominently and to allow disabling it for certain checkable objects. This would be very helpful for exact schedules for a cron-like usage.

[jschmidt-icinga]

As the linked documentation already suggests, maybe this is the time to look at a different algorithm for determining check scheduling.

Ideally the user shouldn't have to override things when our algorithm breaks. And without being completely sure on all the intricacies of the current approach, I think it's safe to say that load-balancing check scheduling via a random number offset is a bit of a brute-force hope-for-the-best approach.

Maybe we could implement some proper load-balancing algorithm that evens out the time-slots, first with a best guess, and without knowing the execution times/latencies, then with smaller and smaller adjustments over time to account for error, like a PID-controller that minimizes uneven spacing between checks, until an equilibrium is reached and checks can be scheduled with more or less their exact check_intervals. This could also give checks scheduled with check_periods priority when the time-slots are (re)calculated so the user gets as close to "expected behavior" as possible, without needing complex documentation to understand what's going on or to know when manual intervention is necessary.

[oxzi]

As the linked documentation already suggests, maybe this is the time to look at a different algorithm for determining check scheduling.

In general, full ACK. And thanks for your ideas. However, I am currently seeing two requirements - potentially conflicting.

1. There should be no unnecessary load spikes. Currently, this is addressed by the scheduling offsets.
2. Users are expecting a cron-like scheduling and using check_interval in this way.

Another idea would be to give each check a randomized startup interval, but schedule them then based on the check_interval. Thus, the checks were kinda balanced first and keep this interval.