Skip to content

No-repeat notification setting bypassed by nflog data retention expiry #10043

@srikanthccv

Description

@srikanthccv

Description

When alerts remain in a firing state for longer than AlertManager's data retention period (default 120h/5 days), users receive notifications even when "no repeat notifications" is configured.

What happens

  1. Alert fires and notification is sent
  2. Alert condition remains true (never resolves)
  3. After 5 days, nflog data retention purges the notification state
  4. AM treats the still-firing alert as "new" and re-sends notifications
  5. "Triggered Alerts" UI shows "firing since" reset to current time
  6. Alert history shows no new entries (because no actual state transition occurred)

Expected behavior

Alerts that have never resolved should not re-fire notifications regardless of internal retention settings.

Observed behavior

Users receive duplicate notifications every ~5 days for long-running alerts. The "firing since" timestamp resets, but alert history remains unchanged.

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions