[RFC] Monitoring rules configuration

[ganto]

As already mentioned in debops/debops-playbooks#160 the application configuration of Check_MK monitoring rules is a bit tricky as the Web interface (WATO) allows the interactive configuration of the entire setup.

From my point of view it should be somehow possible to generate the complete monitoring configuration without manual steps. Especially because Check_MK only offers a rather limited configuration API. So far it's not possible to setup new monitoring rules via an upstream API. I wish this to be possible via Ansible, although the configuration file format is sometimes a bit cumbersome which will make it a bit tricky.

Generally I plan to do the following:

• Allow to configure most variables in the etc/check_mk/conf.d/wato and etc/check_mk/multisite.d/wato via Ansible inventory. This will properly show the configuration in the WATO Web interface, but also allows admins to modify them. However, a subsequent run of Ansible would reset manual configurations in this case again. Manual modifications could be prevented by altering the privileges of the admin role, which could degrade WATO to a mostly read-only interface.
• The same configuration definitions can also be made in the etc/check_mk/conf.d (and etc/check_mk/multisite.d) directories. This is the default location if WATO is not installed. In our case these locations could be used to write configuration definitions which won't clash with manual modifications done through WATO. Unfortunately these definitions also won't be showed by WATO, so it might be trickier to figure out the setup for inexperienced admins.

User of the Ansible role don't have to decide for one or the other, but can set a flag for each configuration variable to indicate where it should be persisted in the file system.

@ypid also mentioned the idea of parsing the configuration files prior to writing it via Ansible, so it would be possible to merge automated and manual configuration rules. This most likely would result in a custom Ansible module which also would need a mechanism to be integrated into DebOps properly. This is definitely a super nice idea but requires a huge amount of development effort. From my point of view I'd rather prefer to see time being spent on a proper upstream configuration API, similar to e.g. what is available with Icinga 2 now.

So, that's my plan for the next few weeks. Let's see how far I can get. If you have some comments, questions, ideas about this topic or generally this Ansible role, let me know.

[ypid]

@ganto Great work!

also mentioned the idea of parsing the configuration files prior to writing it via Ansible

I agree that we can spend our time with more useful things. The WATO Web-API looks like a starting point.

[ganto]

So, it seems that with #22 I completed the first milestone that I set myself:

The role is now able to generate the biggest part of the upstream Check_MK application configuration, especially the monitoring rules. I further established a hopefully comprehensible and flexible mechanism for extending the file templates with the uncountable upstream configuration variables. Examples on how this can be done can be found in the tests directory.

Over time I will try to add all the variables that I currently use in my various Check_MK setups. Still this is a very small number of what is available upstream. Of course PRs are always welcome 😄

Another thing that I haven't really looked at yet is rule ordering...

Still I would be interested in some feedback, what is critical for you that I haven't touched yet?