alertmanager_notification

[Zakariaa222]

Hi,
I configured the alertmanager.yml file to send me alerts to my outlook email box, but I don't receive anything.
To check I typed in the terminal the following command: smtp.office365.com:587 but it does not pass.
What is the solution please

[grobinson-grafana]

Hi! 👋 This isn't enough information to be able to help I'm afraid. Perhaps you can share the logs from your Alertmanager when running with --log.level=debug?

[Zakariaa222]

Hi,
I added the line --log.level=debug in the file "/etc/systemd/system/alertmanager.service" like this :

[Unit]
Description=AlertManager for Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager --config.file /etc/alertmanager/alertmanager.yml --log.level=debug

[Install]
WantedBy=multi-user.target

After i typed the command journalctl -u alertmanager in the terminal to get the Alertmanager logs and here is the result:

juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.071Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revi>
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.071Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:5>
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.071Z caller=cluster.go:173 level=debug component=cluster msg="resolved peers to following addresses" peers=
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.072Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=19>
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.073Z caller=delegate.go:236 level=debug component=cluster received=NotifyJoin node=01H2DKB1B87YGSEVCY51YFT>
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.073Z caller=cluster.go:266 level=debug component=cluster msg="joined cluster" peers=0
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.074Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.124Z caller=main.go:395 level=debug externalURL=http://prometheus:9093
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.124Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/e>
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.125Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration >
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.129Z caller=main.go:441 level=info component=configuration msg="skipping creation of receiver not referenc>
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.129Z caller=main.go:525 level=debug routePrefix=/
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.131Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093
juin 08 14:32:36 prometheus alertmanager[283423]: ts=2023-06-08T13:32:36.131Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093
juin 08 14:32:38 prometheus alertmanager[283423]: ts=2023-06-08T13:32:38.074Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 el>
juin 08 14:32:40 prometheus alertmanager[283423]: ts=2023-06-08T13:32:40.074Z caller=cluster.go:703 level=debug component=cluster msg="gossip looks settled" elapsed=4.0006136s
juin 08 14:32:42 prometheus alertmanager[283423]: ts=2023-06-08T13:32:42.075Z caller=cluster.go:703 level=debug component=cluster msg="gossip looks settled" elapsed=6.0013831s
juin 08 14:32:44 prometheus alertmanager[283423]: ts=2023-06-08T13:32:44.075Z caller=cluster.go:703 level=debug component=cluster msg="gossip looks settled" elapsed=8.0017764s

[grobinson-grafana]

Great! You also need to send an alert to the Alertmanger so it attempts to send an email using the configured smtp configuration, otherwise we won't be able to see why it failed.

[Zakariaa222]

I have an ubuntu machine which I typed in its terminal the following command stress --cpu 4 to increase the cpu usage to 100% so the alertmanager alert will trigger.
Here is the alert in the file alert_rules.yml :

• alert: HighCPUUsage_Linux
  expr: (sum by(instance) (irate(node_cpu_seconds_total{mode!="idle"}[1m])) / on(instance) group_left sum by (instance)((irate(node_cpu_seconds_total{}[1m])))) * 100 > 90
  for: 1m
  labels:
  severity: critical
  annotations:
  description: ATTENTION, CPU usage on {{ $labels.instance }} is above 90% for more than 1 minute

And here is the alertmanager.yml file which contains the smtp configuration :

global:

route:
group_by: ['alertname']
receiver: 'email_outlook'

• name: 'email_outlook'
  email_configs:
  • to: "zakariaa.@outlook.com"
    from: "zakariaa.@outlook.com"
    smarthost: "smtp.office365.com:587"
    auth_username: "zakariaa.@outlook.com"
    auth_password: "****"