[Bug] ContactPoint with valuesFrom Secret always triggers an update: Grafana returns [REDACTED] for secret fields

[YuriBucci-Solfacil]

[Bug] ContactPoint with valuesFrom Secret always triggers an update: Grafana returns [REDACTED] for secret fields

Describe the bug

A GrafanaContactPoint that resolves any of its settings via valuesFrom.secretKeyRef is reconciled as "drifted" on every reconciliation period, even when neither the CR nor the remote contact point have changed. The log line below is emitted every resyncPeriod:

info  GrafanaContactPointReconciler  update existing contact point receiver  {"uid": "..."}

Root cause: Grafana's /api/v1/provisioning/contact-points endpoint redacts secret fields in the response body (returning the literal string "[REDACTED]"), but the operator compares those redacted values byte-for-byte against the actual resolved secret it would send on a PUT.

Evidence from the code

controllers/contactpoint_controller.go does:

remote := remoteReceivers[existingIdx]
if cp.Name != remote.Name ||
    *cp.Type != *remote.Type ||
    cp.DisableResolveMessage != remote.DisableResolveMessage ||
    !reflect.DeepEqual(cp.Settings, remote.Settings) {   // <-- always different when secrets are used
    log.Info("update existing contact point receiver", "uid", recUID)
    // PUT
}

cp.Settings is built by buildContactPointSettings, which resolves every valuesFrom.SecretKeyRef into the real secret value:

val, _, err := getReferencedValue(ctx, r.Client, cr.Namespace, override.ValueFrom)
...
simpleContent.SetPath(strings.Split(override.TargetPath, "."), val)

Whereas remote.Settings comes from GetContactpoints which, since Grafana 8.x+, masks any field flagged as secret and returns "[REDACTED]" in its place.

Empirical reproduction

With the CR below:

apiVersion: grafana.integreatly.org/v1beta1
kind: GrafanaContactPoint
metadata:
  name: my-slack
spec:
  instanceSelector:
    matchLabels:
      dashboards: grafana
  name: my-slack
  type: slack
  settings:
    recipient: "#my-channel"
  valuesFrom:
    - targetPath: token
      valueFrom:
        secretKeyRef:
          name: my-secret
          key: SLACK_TOKEN

After the operator creates the contact point, calling the Grafana API directly returns:

{
  "name": "my-slack",
  "type": "slack",
  "settings": {
    "recipient": "#my-channel",
    "token": "[REDACTED]"
  },
  "provenance": "api"
}

But cp.Settings on the operator side is:

{
  "recipient": "#my-channel",
  "token": "xoxb-<actual-slack-token>"
}

reflect.DeepEqual therefore returns false on every reconcile, causing an unconditional PUT and unnecessary load on Grafana. With defaultResyncPeriod: 1m this amounts to 60 writes per hour per contact point with secrets.

Version

• grafana-operator: v5.22.2
• Grafana: 12.1.0 (external mode)

To reproduce

1. Deploy grafana-operator.
2. Create any GrafanaContactPoint that uses valuesFrom with a secretKeyRef for fields such as token, password, url, webhook, apiKey, etc.
3. Watch operator logs: "update existing contact point receiver" is emitted on every resyncPeriod forever.

Expected behavior

When the CR has not changed and the remote contact point also has not changed, no PUT should be issued.

Proposed fix

The comparison step in reconcileWithInstance needs to ignore [REDACTED] fields on the remote side. A pragmatic approach:

• While diffing cp.Settings against remote.Settings, any field whose remote value equals the string "[REDACTED]" should be considered "unchanged" regardless of the local value. If every other field matches, skip the PUT.

Alternatively, maintain a hash of the rendered cp.Settings in cr.Status.Hash (similar to what GrafanaFolder does) and use that to detect true drift on the CR side, without relying on the redacted GET response.

Suspect component/Location

• https://github.com/grafana/grafana-operator/blob/v5.22.2/controllers/contactpoint_controller.go#L224-L241

Runtime

• Environment: AWS EKS running Kubernetes v1.34
• External Grafana: v12.1.0
• Deployment type: Helm chart

[Baarsgaard]

I'm quite hesitant to track this outside of Grafana as there's multiple ways to end up in an infinite "It matches" loop if a single Instance fell out or was unhealthy for a while and the state was tracked in .status.
Which could result in a ContactPoint not being correctly provisioned, resulting in an alert not being sent.

Unless we find a way for Grafana to hold and confirm its last provisioned state, either directly or via metadata, I'd instead change the check to never skip reconciliation when a secret is referenced in any .spec.valuesFrom[] entry.

[theSuess]

Talked about this with @Baarsgaard in the weekly sync. We decided to go with an unconditional reconcile (=always update the resource). Shouldn't cause any extra load, we can simplify our logic and we need one less network call