Alerts fail with "stage for receiver missing" error - default-receiver has no webhook config

[gwilliamspro]

Version

v0.108.0 (latest release)

Deployment

Docker Compose (self-hosted)

---

Error Message

{"level":"ERROR","msg":"Notify for alerts failed","component":"signoz-dispatcher","num_alerts":1,"err":"stage for receiver missing"}

Location: pkg/alertmanager/alertmanagerserver/dispatcher.go:337

---

Problem Summary

Alert notifications fail because:

1. SigNoz generates an alertmanager_config with default-receiver having no webhook_configs
2. When alerts fire, the dispatcher tries to notify default-receiver but fails because there's no notification stage

---

Steps to Reproduce

1. Fresh SigNoz v0.108.0 install via Docker Compose
2. Create a webhook notification channel (e.g., "discord-ops")
3. Create alert rules with that channel assigned
4. Wait for alerts to fire (or trigger manually from UI)
5. Real alerts fail with "stage for receiver missing"
6. Note: Test notifications from the UI work fine

---

Generated alertmanager_config (from SQLite)

{
  "route": {
    "receiver": "default-receiver",
    "routes": [{
      "receiver": "discord-ops",
      "matchers": ["ruleId=~\"...|...|...\""],
      "continue": true
    }]
  },
  "receivers": [
    {"name": "default-receiver"},
    {"name": "discord-ops", "webhook_configs": [...]}
  ]
}

Issue: default-receiver is empty - no webhook_configs array.

---

Notification Channel (from SQLite)

id: 019bf832-9ba4-78e8-8b3c-4e0eb61f573c
name: discord-ops
type: webhook
data: {"name":"discord-ops","webhook_configs":[...]}

---

Alert Rules

• 12 rules configured
• All assigned to channel via preferredChannels
• Example rule has "preferredChannels": ["discord-ops"]

---

Config Overwrite Behavior

Attempting to manually fix the database config results in SigNoz detecting a "mismatch" and overwriting it:

{"msg":"mismatch found, updating config to match channels and matchers"}

This happens in pkg/alertmanager/service.go:250 (compareAndSelectConfig)

---

Log Sequence

05:03:56 - "number of alerts found", rule_name="High Error Rate", alerts_count=1
05:03:52 - ERROR "Notify for alerts failed", err="stage for receiver missing"
05:04:22 - ERROR "Notify for alerts failed", err="stage for receiver missing"
05:09:22 - ERROR "Notify for alerts failed", err="stage for receiver missing"

---

Expected Behavior

• Alerts matching a route should be delivered to the configured webhook
• If default-receiver must exist, it should have a valid (or no-op) notification stage

Actual Behavior

• Alerts fail with "stage for receiver missing"
• The default-receiver is created with no webhook config
• Manual DB fixes are overwritten by compareAndSelectConfig

---

Workaround Attempted

• Patching alertmanager_config directly in SQLite to add webhook to default-receiver
• Setting continue: false on routes
• Result: SigNoz detects "mismatch" and overwrites on next config sync

---

Environment

• Docker Compose deployment
• SigNoz image: signoz/signoz:v0.108.0
• Ubuntu 24.04 host
• ClickHouse for telemetry storage

[welcome]

Thanks for opening this issue. A team member should give feedback soon. In the meantime, feel free to check out the contributing guidelines.