Reduce Telegram/provider alert floods during transient DNS/registry outages

[RaviTharuma]

Describe the bug / problem

Operators who wire notification-controller to chat providers (Telegram, Slack, etc.) can receive a flood of error events when cluster DNS or an external registry is briefly unavailable (e.g. a short CoreDNS blip, CNI hiccup, or registry connectivity flap).

Within a few seconds, many Flux objects fail in parallel and each emits its own error event. Typical patterns observed in a homelab-style cluster:

Source kind	Example failure class (message shape)
GitRepository	checkout / clone failed: dial tcp: lookup …: connection refused / temporary DNS failure
HelmRepository	index / chart fetch failed with the same transient network/DNS errors
ImageRepository	scan failed: dial/lookup connection refused or temporary failure
ImagePolicy	no tags in database / empty version list (often cascading from a failed scan)

With a Telegram (or similar) Provider, this becomes dozens of messages in a short window for what is essentially one transient infrastructure incident—not dozens of distinct actionable failures.

This is especially noisy when:

• Many sources share the same DNS path (cluster CoreDNS → upstream)
• Image reflection is enabled for multiple repositories
• Alerts select broad eventSources (e.g. name: '*') with eventSeverity: error or info

What already exists (and why it is not enough alone)

1. Alert.spec.exclusionList — message-regex filtering works, but operators must reverse-engineer regexes for every controller’s wording. Related: Document exclusionList examples to ignore commonly spammy notifications #283 (document common exclusion examples), Allow filtering Alerts by event reason, not just message #1359 (filter by event reason, not only message).
2. Controller flag --rate-limit-interval (default 5m) — useful for identical repeated events on the same condition, as discussed in Alerts duplicate error messages in slack channel #445. It does not stop a cascade across many different objects during a cluster-wide DNS/registry blip: each object has a distinct involved object / message, so each still pages the chat.

Steps to reproduce

1. Configure a Telegram (or Slack) Provider + Alert covering many sources, e.g.:

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: flux-system
  namespace: flux-system
spec:
  providerRef:
    name: telegram
  eventSeverity: error
  eventSources:
    - kind: GitRepository
      name: '*'
    - kind: HelmRepository
      name: '*'
    - kind: ImageRepository
      name: '*'
    - kind: ImagePolicy
      name: '*'

2. Briefly break cluster DNS or outbound registry access (e.g. CoreDNS unavailable for a short window, or temporary block to example.com registry).
3. Observe a burst of error events across Git/Helm/Image objects, each delivered as a separate chat message within seconds.

Expected behavior

For retryable / transient network and DNS failures, operators should be able to:

1. Documented, copy-pasteable exclusionList patterns for common transient network/DNS messages (and guidance on what is safe vs dangerous to exclude).
2. Optionally rate-limit, debounce, or aggregate notifications per object (or per outage window) on the Provider or Alert CR—so one DNS blip does not page once per object.
3. Clarity on whether ImageRepository / ImagePolicy should default to less noisy event severity (or suppress intermediate states) for retryable scan failures and downstream “no tags in database” / empty version-list errors, vs treating them the same as hard permanent failures.

Proposed product improvements (discussion welcome)

These are product ideas, not a request for private config review.

1. Document recommended exclusionList patterns for transient network/DNS errors

Extend Alert docs (and finish the spirit of #283) with tested regex examples for transient failures, e.g. (illustrative only—wording should match actual controller messages):

spec:
  exclusionList:
    # Transient DNS / dial failures from sources (tune to actual message text)
    - ".*dial tcp:.*lookup .*: (connection refused|i/o timeout|temporary failure|server misbehaving).*"
    - ".*no such host.*"
    # Cascading image policy noise after a failed scan (if operators choose to suppress)
    - ".*no tags in database.*"

Docs should cover:

• Safe defaults vs “you may hide real outages”
• That message wording can change across controller versions (link to Allow filtering Alerts by event reason, not just message #1359 for reason-based filtering)
• Interaction with existing global --rate-limit-interval

2. Optional event rate-limit / debounce / aggregate on Provider or Alert

Today rate limiting is a global controller flag. A CR-level control would help multi-tenant and chat-noise use cases without changing the default for everyone, for example:

# Illustrative API — not a concrete proposal
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: flux-system
spec:
  # debounce identical (or same involved-object) errors for a window
  eventRateLimit:
    interval: 15m
    # optional: collapse N failures of the same class into one summary message
    aggregate: true

Useful behaviors to consider:

• Key by involved object (+ severity/reason), not only full message text
• Aggregate burst: “12 sources failed with DNS lookup errors in 30s” instead of 12 messages
• Per-Provider vs per-Alert configuration (chat bots vs webhooks may want different policies)

3. Less noisy defaults for retryable ImageRepository / ImagePolicy scan failures

Cross-controller (image-reflector-controller / event emission), but it drives notification spam:

• Should retryable scan failures (dial/lookup refused, temporary DNS) emit at a lower severity, or only surface after N consecutive failures / non-retryable terminal state?
• Should ImagePolicy “no tags in database” / empty version list be info/debug or suppressed while the parent ImageRepository is still retrying, and only alert when the policy is stuck beyond a threshold?

Happy to split (3) into an issue against image-reflector-controller if maintainers prefer.

Related issues

• Document exclusionList examples to ignore commonly spammy notifications #283 — Document exclusionList examples for spammy notifications (open; docs bank of regexes)
• Allow filtering Alerts by event reason, not just message #1359 — Filter Alerts by event reason, not only message
• Alerts duplicate error messages in slack channel #445 — Duplicate error messages / existing --rate-limit-interval behavior
• ImagePolicy does not reconcile after "no tags found in database" error flux2#5587 — ImagePolicy / “no tags found in database” reconcile behavior (related cascade)

Environment (generic)

• Flux notification-controller with Telegram (or other chat) Provider
• Homelab / multi-source cluster: many GitRepository, HelmRepository, ImageRepository, ImagePolicy objects
• Transient CoreDNS or registry connectivity outages (seconds to a few minutes)

Additional context

Goal is actionable alerts: permanent misconfiguration and sustained outages should still page; short-lived infrastructure blips should not dominate the chat channel. Documentation + optional CR-level debouncing would make Flux much friendlier for operators using direct chat providers without an intermediate Alertmanager.

I am happy to help with a docs PR for (1) once maintainers confirm which message patterns are considered stable enough to publish.

[RaviTharuma]

Still needed on notification-controller v1.9.2 (2026-08)

Following up: this remains an operational gap on v1.9.2.

• Alert.spec.exclusionList is still the primary lever against transient DNS/registry and cascading image-automation noise.
• Alert and Provider still expose no useful status (status: {}), so we cannot see from the CR how many events were excluded vs sent, nor the last dispatch error (401/429/network). Debugging still means controller logs.
• Global --rate-limit-interval still does not stop a multi-object cascade during one short outage (each object is a distinct event).

Related follow-ups filed separately

• Observability of Alert/Provider status conditions (last dispatch / last error) — distinct from flood volume; will reference this issue.
• Image automation cold-start event spam remains active on image-reflector-controller v1.2.3 (no tags in database until scans refill); exclusionList examples still belong in docs (build(deps): bump github/codeql-action from 3.26.4 to 3.26.9 in the ci group across 1 directory #930 and a website docs request).

Ask (unchanged priorities)

1. Documented, copy-pasteable exclusionList patterns for transient network/DNS + image cold-start messages.
2. Optional CR-level debounce/aggregate so one infrastructure blip is not one chat message per object.
3. Longer term: status on Alert/Provider so operators can validate that exclusions and dispatches are doing what they think.

Environment: notification-controller v1.9.2, Alert v1beta3 eventSeverity: error, Provider type telegram (details omitted), Flux distribution flux-v2.9.1 / CLI v2.9.4.