Lower alert-logger route repeat interval for watchdog alert

[alexandre-allard]

Component: salt, monitoring

Why this is needed:
In MetalK8s UI, we need to display the monitoring pipeline status.
For this, we check the Watchdog alert, but this alert is only fired once every 12h, meaning we can end up with a broken monitoring for few hours before noticing it.

What should be done:
We should reduce the interval between sending 2 Watchdog alerts.

Implementation proposal (strongly recommended):
In salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml, we must create a new route to handle only the Watchdog alert and lower the repeat interval to something like 1m, e.g:

diff --git a/salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml b/salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml
index 29f12b93f..dee365c03 100644
--- a/salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml
+++ b/salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml
@@ -19,6 +19,12 @@ spec:
         repeat_interval: 12h
         receiver: 'metalk8s-alert-logger'
         routes:
+        - receiver: 'metalk8s-alert-logger'
+          group_interval: 1m
+          repeat_interval: 1m
+          match:
+            alertname: Watchdog
+          continue: True
         - receiver: 'metalk8s-alert-logger'
           continue: True
       receivers:

Bonus, we could also lower the global repeat interval as the actual value is a bit high:

diff --git a/salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml b/salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml
index 29f12b93f..196a34fe1 100644
--- a/salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml
+++ b/salt/metalk8s/addons/prometheus-operator/config/alertmanager.yaml
@@ -16,7 +16,7 @@ spec:
         group_by: ['job']
         group_wait: 30s
         group_interval: 5m
-        repeat_interval: 12h
+        repeat_interval: 1h
         receiver: 'metalk8s-alert-logger'
         routes:
         - receiver: 'metalk8s-alert-logger'

Test plan: