OBSDOCS-1327: Improve troubleshooting monitoring issues: new section troubleshooting alertmanager configurations #92246
Conversation
…troubleshooting alertmanager configurations
openshift-ci-robot
added
the
jira/valid-reference
|
@eromanova97: This pull request references OBSDOCS-1327 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.19.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
openshift-ci
bot
added
the
size/L
|
@eromanova97: all tests passed! Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
@eromanova97: This pull request references OBSDOCS-1327 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.19.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
LGTM, waiting for others to review |
|
@eromanova97: This pull request references OBSDOCS-1327 which is a valid jira issue. Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the bug to target the "4.19.0" version, but no target version was set. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
| [id="troubleshooting-alertmanager-configurations_{context}"] | ||
| = Troubleshooting Alertmanager configuration | ||
|
|
||
| If your Alertmanager configuration does not work properly, you can compare the `alertmanager-main` secret with the running Alertmanager configuration to identify possible errors. You can also test your alert routing configuration by creating a test alert. |
There was a problem hiding this comment.
I'd like to learn more about those errors? are we talking about cases where the user breaks the config and Alertmanager cannot/doesn't load it?
There was a problem hiding this comment.
isn't AlertmanagerFailedReload triggered in this case? maybe we can have this in https://github.com/openshift/runbooks/blob/407f97961c22c72f57edf599efe95eadd6baf780/alerts/cluster-monitoring-operator/AlertmanagerFailedReload.md?plain=1#L2?
There was a problem hiding this comment.
see https://github.com/openshift/openshift-docs/pull/92246/files#r2050261250, mail_configs, smarthost missed port, for this case, AlertmanagerFailedReload alert would be fired
if set smarthost to a unreachable value, example
receivers:
- name: 'web.hook'
email_configs:
- to: ***
from: ***
smarthost: 'smtp.non-exist.com:25'
AlertmanagerFailedToSendAlerts would be fired, AlertmanagerFailedReload would not be fired
# token=`oc create token prometheus-k8s -n openshift-monitoring`
# oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -H "Authorization: Bearer $token" 'https://thanos-querier.openshift-monitoring.svc:9091/api/v1/query?' --data-urlencode 'query=count (ALERTS{alertname=~"AlertmanagerFailedReload|AlertmanagerFailedToSendAlerts"}) by (alertname)' | jq
{
"status": "success",
"data": {
"resultType": "vector",
"result": [
{
"metric": {
"alertname": "AlertmanagerFailedToSendAlerts"
},
"value": [
1744962937.043,
"2"
]
}
],
"analysis": {}
}
}
error in alertmanager pod logs
time=2025-04-18T07:46:52.334Z level=ERROR source=dispatch.go:360 msg="Notify for alerts failed" component=dispatcher num_alerts=1 err="web.hook/email[0]: notify retry canceled after 7 attempts: establish connection to server: dial tcp: lookup smtp.non-exist.com on 172.30.0.10:53: no such host"
I think we could mention AlertmanagerFailedToSendAlerts and AlertmanagerFailedReload or just mention to check any alerts related to Alertmanager
There was a problem hiding this comment.
@machine424 @juzhao I could add step 1 like this: WDYT?
. Check active alerts related to Alertmanager:
+
[source,terminal]
----
$ oc exec alertmanager-main-0 -n openshift-monitoring -- amtool alert --alertmanager.url http://localhost:9093
----
+
.Example output
[source,terminal]
----
Alertname Starts At Summary State
Watchdog 2025-04-28 08:01:41 UTC An alert that should always be firing to certify that Alertmanager is working properly. active
AlertmanagerFailedToSendAlerts 2025-04-28 08:11:54 UTC An Alertmanager instance failed to send notifications. active <1>
----
<1> Look for alerts that indicate an issue with Alertmanager, such as `AlertmanagerFailedToSendAlerts` or `AlertmanagerFailedReload`.
.. If you identified an alert related to Alertmanager, list the alert's runbook URL:
+
.Example command
[source,terminal]
----
$ oc get prometheusrules -n openshift-monitoring -o yaml | grep 'AlertmanagerFailedToSendAlerts' | grep 'runbook_url'
----
+
.Example output
[source,terminal]
----
runbook_url: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/AlertmanagerFailedToSendAlerts.md
----
.. Open the rubook URL and follow the instructions described in the runbook.
There was a problem hiding this comment.
I am not sure if this is an overkill or not 😄 but we do not really mention the existence of runbooks anywhere in monitoring docs, so maybe having this here could be useful.
Or, I could plan to have something like the virtualization team has:
https://docs.redhat.com/en/documentation/openshift_container_platform/4.18/html/virtualization/monitoring#virt-runbooks
There was a problem hiding this comment.
Yes, the alerts should have links to their runbooks.
I suggest we enrich runbooks as users will mainly be notified about these issues via alerts.
Also that will avoid having to duplicate and maintain the troubleshooting in two different places.
(we could consider/discuss having links to the runbooks in the docs, yes)
Let's mention in AlertmanagerFailedReload runbook that one should look for logs with "Loading configuration file failed" (as shown here https://github.com/openshift/openshift-docs/pull/92246/files#r2050261250), sth like:
$ NAMESPACE='<value of namespace label from alert>'
$ oc -n $NAMESPACE logs -l 'app.kubernetes.io/name=alertmanager' --tail=-1 | \
grep 'Loading configuration file failed.*' \
| sort | uniq -c | sort -n
time=2025-04-18T07:28:00.274Z level=ERROR source=coordinator.go:117 msg="Loading configuration file failed" component=configuration file=/etc/alertmanager/config_out/alertmanager.env.yaml err="address smtp.gmail.com: missing port in address"
as we do in other runbooks, in https://github.com/openshift/runbooks/blob/f31c57f491b68b07ad6d1a39d45189bd780be8a7/alerts/cluster-monitoring-operator/PrometheusRuleFailures.md e.g.
note that the alert could be triggered for the platform or the uwm alertmanager.
we can say that the err field should help locate the issue.
There was a problem hiding this comment.
we can do the same for AlertmanagerFailedToSendAlerts's runbook with the grep on Notify for alerts failed + add the guide for how to send a test alert to a receiver as that could help reproduce the issue and assist with diagnostics.
| <1> The example shows the route to the `team-frontend-page` receiver. Alertmanager routes alerts with `service="example-app"` and `severity="critical"` labels to this receiver. | ||
| <2> The `team-frontend-page` receiver configuration. The example shows PagerDuty as a receiver. | ||
|
|
||
| .. Compare the contents of the `route` and `receiver` fields of the `alertmanager.yaml` file with the fields in the running Alertmanager configuration. Look for any discrepancies. |
There was a problem hiding this comment.
comparing could be tedious, maybe we should think about a diff command or sth.
If we only suspect the secret to not be taken into account by AM, maybe checking a log somewhere after changing the secret is sufficient?
There was a problem hiding this comment.
@machine424 Note: I asked the team during the office hours, and there does not seem to be a diff command that cames to mind right away, so if we do not come up with something, I will concider this as possible improvement that can be added later 👍
|
thanks for this Eliska, |
|
After the discussions, I will be closing the PR as the enhancements will be done in other parts of our docs experience. |
Version(s): 4.12 and later
Issue: https://issues.redhat.com/browse/OBSDOCS-1327
Link to docs preview: https://92246--ocpdocs-pr.netlify.app/openshift-enterprise/latest/observability/monitoring/troubleshooting-monitoring-issues.html#troubleshooting-alertmanager-configurations_troubleshooting-monitoring-issues
QE review:
Additional information: