Skip to content

[kube-prometheus-stack] Alert manager not sending to MS-teams #5345

Open
@alshoaili96

Description

@alshoaili96

Describe the bug a clear and concise description of what the bug is.

I am trying to send Alertmanager notifiation to a teams channel, however with the configuration i have i am getting this error:
time=2025-02-24T08:25:42.616Z level=ERROR source=dispatch.go:360 msg="Notify for alerts failed" component=dispatcher num_alerts=1 err="MS-TEAMS/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: Summary or Text is required." time=2025-02-24T08:26:12.980Z level=ERROR source=dispatch.go:360 msg="Notify for alerts failed" component=dispatcher num_alerts=1 err="MS-TEAMS/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: Summary or Text is required."

and this is my current configuration:

config:
    global:
      resolve_timeout: 5m
    inhibit_rules: []
    route:
      group_by: ['namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'MS-TEAMS' #original value is 'null'
      routes: []
    receivers:
    - name: 'null'
    - name: 'MS-TEAMS'
      msteams_config:
        - send_resolved: true
          webhook_url: "webhook_url"
          title: "{{ template \"msteams.default.title\" . }}"  # Default title template
          summary: "{{ .CommonAnnotations.summary | default \"No summary available\" }}"  # Ensures a default value
          text: "{{ .CommonAnnotations.description | default .CommonAnnotations.summary }}"  # Fallback to summary if description is missing
          max_alerts: 1  # Optional, sends alerts individually, remove later
    templates:
    - '/etc/alertmanager/config/*.tmpl'

following this documentation on alertmanager-msteams-config:
https://prometheus.io/docs/alerting/latest/configuration/#msteams_config

tried to do a curl to test:
curl -XPOST -H "Content-Type: application/json" -d '[ { "labels": { "alertname": "TestAlert", "severity": "critical" }, "annotations": { "summary": "Manual test alert for Microsoft Teams", "description": "If you see this message in Teams, the webhook is working!" } } ]' http://localhost:9093/api/v2/alerts
which gives an empty response and i see the alert in alertmanager but not in teams, so i tested the webhook to check if it is working:
curl -XPOST -H "Content-Type: application/json" -d '{ "title": "Teams Webhook Test", "text": "If you see this in Teams, the webhook is working!" }'webhook-url
with this i can see the message in the teams channel.

appreciate all the help and support

What's your helm version?

version.BuildInfo{Version:"v3.17.1", GitCommit:"980d8ac1939e39138101364400756af2bdee1da5", GitTreeState:"clean", GoVersion:"go1.23.6"}

What's your kubectl version?

Client Version: v1.31.1 Kustomize Version: v5.4.2 Server Version: v1.30.3+k0s

Which chart?

kube-prometheus-stack

What's the chart version?

latest

What happened?

I am trying to send Alertmanager notifiation to a teams channel, however with the configuration i have i am getting this error:
time=2025-02-24T08:25:42.616Z level=ERROR source=dispatch.go:360 msg="Notify for alerts failed" component=dispatcher num_alerts=1 err="MS-TEAMS/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: Summary or Text is required." time=2025-02-24T08:26:12.980Z level=ERROR source=dispatch.go:360 msg="Notify for alerts failed" component=dispatcher num_alerts=1 err="MS-TEAMS/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: Summary or Text is required."

and this is my current configuration:

config:
    global:
      resolve_timeout: 5m
    inhibit_rules: []
    route:
      group_by: ['namespace']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 12h
      receiver: 'MS-TEAMS' #original value is 'null'
      routes: []
    receivers:
    - name: 'null'
    - name: 'MS-TEAMS'
      msteams_config:
        - send_resolved: true
          webhook_url: "webhook_url"
          title: "{{ template \"msteams.default.title\" . }}"  # Default title template
          summary: "{{ .CommonAnnotations.summary | default \"No summary available\" }}"  # Ensures a default value
          text: "{{ .CommonAnnotations.description | default .CommonAnnotations.summary }}"  # Fallback to summary if description is missing
          max_alerts: 1  # Optional, sends alerts individually, remove later
    templates:
    - '/etc/alertmanager/config/*.tmpl'

following this documentation on alertmanager-msteams-config:
https://prometheus.io/docs/alerting/latest/configuration/#msteams_config

tried to do a curl to test:
curl -XPOST -H "Content-Type: application/json" -d '[ { "labels": { "alertname": "TestAlert", "severity": "critical" }, "annotations": { "summary": "Manual test alert for Microsoft Teams", "description": "If you see this message in Teams, the webhook is working!" } } ]' http://localhost:9093/api/v2/alerts
which gives an empty response and i see the alert in alertmanager but not in teams, so i tested the webhook to check if it is working:
curl -XPOST -H "Content-Type: application/json" -d '{ "title": "Teams Webhook Test", "text": "If you see this in Teams, the webhook is working!" }'webhook-url
with this i can see the message in the teams channel.

appreciate all the help and support

What you expected to happen?

recieve an alert in my teams channel

How to reproduce it?

No response

Enter the changed values of values.yaml?

config:
global:
resolve_timeout: 5m
inhibit_rules: []
route:
group_by: ['namespace']
group_wait: 30s
group_interval: 5m
repeat_interval: 12h
receiver: 'MS-TEAMS' #original value is 'null'
routes: []
receivers:
- name: 'null'
- name: 'MS-TEAMS'
msteams_config:
- send_resolved: true
webhook_url: "webhook_url"
title: "{{ template "msteams.default.title" . }}" # Default title template
summary: "{{ .CommonAnnotations.summary | default "No summary available" }}" # Ensures a default value
text: "{{ .CommonAnnotations.description | default .CommonAnnotations.summary }}" # Fallback to summary if description is missing
max_alerts: 1 # Optional, sends alerts individually, remove later
templates:
- '/etc/alertmanager/config/*.tmpl'

Enter the command that you execute and failing/misfunctioning.

helm

Anything else we need to know?

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions