prometheus-operator StatefulSet does not rollout all pods

[angelbarrera92]

Hi team!

We are facing something similar to #145

The reloader works fine for regular statefulsets but something is not working correctly with statefulsets managed by prometheus-operator.

Reloader detects the change correctly, but it just triggers the update of one out of N (3 in my case). I'm almost sure this is an incompatibility with Prometheus Operator, WDYT?

Did you face this before?

Thanks

[faizanahmad055]

I do not think this has anything to do with the reloader itself. Reloader just updates the statefulset so based on how the statefulset is configured, it will reload/recreate the pods.

[karl-johan-grahn]

@angelbarrera92 can you provide more detailed reproduction steps?

[cowjen01]

Hello @karl-johan-grahn!

We have a secret called prometheus-nodes-tls with two keys tls.key and tls.crt. We want to watch for changes of this secret (typically the certificate expire and it is automatically refreshed) and restart the prometheus/alertmanager services to load the secret.

Prometheus operator (version 40.x) values.yaml:

prometheus:
  annotations:
    "secret.reloader.stakater.com/reload": "prometheus-nodes-tls"
  serviceMonitor:
    scheme: https
    tlsConfig:
      insecureSkipVerify: true
  prometheusSpec:
    replicas: 2
    scrapeTimeout: 30s
    web:
      tlsConfig:
        clientAuthType: "NoClientCert"
        keySecret:
          name: "prometheus-nodes-tls"
          key: "tls.key"
        cert:
          secret:
            name: "prometheus-nodes-tls"
            key: "tls.crt"
    alertingEndpoints:
      - name: "prometheus-stack-alertmanager"
        namespace: "prometheus-stack"
        port: "http-web"
        scheme: "https"
        tlsConfig:
          caFile: "/mnt/secrets-store/trustedCert"
          serverName: "prometheus-stack-alertmanager"
        apiVersion: v2
        pathPrefix: "/"
    volumes:
      - name: prometheus-secrets-store
        csi:
          driver: secrets-store.csi.k8s.io
          readOnly: true
          volumeAttributes:
            secretProviderClass: prometheus-vault
    volumeMounts:
      - name: prometheus-secrets-store
        mountPath: "/mnt/secrets-store"
        readOnly: true
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: openebs-hostpath
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 2Gi

Pod-reloader values.yaml:

reloader:
  ignoreSecrets: false
  ignoreConfigMaps: false
  reloadStrategy: "annotations"
  enableHA: true
  deployment:
    replicas: 3
  serviceMonitor:
    enabled: true
  podDisruptionBudget:
    enabled: true
    minAvailable: 1

The problem is that just one of the 2 replicas is reloaded.

[faizanahmad055]

@cowjen01 Can you please share reloader logs for the restart? Reloader itself doesn't control the reload strategy of the deployment/statefulset. It will just update either the annotation or environment-variable in deployment/statefulset. You can test this by editing the annotation or environment-variable in the deployment/statefulset and see how it restarts the pods.

[yyardeng]

hello, i am running into the same issue - reloader version 0.0.124, prometheus operator version 0.71.2.
i have the following annotation on the prometheus statefulset (managed by the prometheus operator) to reload the prometheus pods when the secret containing the thanos object storage information is updated:
secret.reloader.stakater.com/reload: "thanos-objstore-config"

but when the secret is updated, only one of the 2 statefulset pods gets reloaded. other statefulsets, not managed by prometheus operator, are reloaded as usual, so i am assuming the problem lies in some incompatibility between the two.

the update strategy defined for the prometheus statefulset is RollingUpdate.

reloader helm values include:

reloader:
isArgoRollouts: false
isOpenshift: false
ignoreSecrets: false
ignoreConfigMaps: false
reloadOnCreate: false
reloadStrategy: default
ignoreNamespaces: ""
logFormat: ""
watchGlobally: true

reloader logs:
time="2024-02-18T14:51:01Z" level=info msg="Changes detected in 'thanos-objstore-config' of type 'SECRET' in namespace 'monitoring', Updated 'prometheus-kube-prometheus-stack-prometheus' of type 'StatefulSet' in namespace 'monitoring'"

[sarabkasada]

Hi All,

I'm running into similar issue where only one of the N pod(s) managed by Prometheus-Operator gets replaced.

Reloader version: v1.0.119
PrometheusOperator version: v0.75.2.

@faizanahmad055 : Please find the response inline

Can you please share reloader logs for the restart?

[Sarab]: Get only the following entry at log when a change in detected in the tracked secret.
time="2024-10-07T05:09:24Z" level=info msg="Changes detected in 'thanos-sidecar-secure-web' of type 'SECRET' in namespace 'monitoring', Updated 'prometheus-kube-prometheus-stack-prometheus ' of type 'StatefulSet' in namespace 'monitoring'"

StatefulSet status when Reloader triggers the update. It just replaces one of the pod and that's it.

Events:
  Type    Reason            Age                   From                    Message
  ----    ------            ----                  ----                    -------
  Normal  Reloaded          36m (x7 over 3h15m)   reloader-secrets        Changes detected in 'thanos-sidecar-secure-web' of type 'SECRET' in namespace 'monitoring', Updated 'prometheus-kube-prometheus-stack-promet
heus' of type 'StatefulSet' in namespace 'monitoring'
  Normal  SuccessfulDelete  13m (x15 over 3h15m)  statefulset-controller  delete Pod prometheus-kube-prometheus-stack-prometheus-1 in StatefulSet prometheus-kube-prometheus-stack-prometheus successful
  Normal  SuccessfulCreate  13m (x16 over 3h15m)  statefulset-controller  create Pod prometheus-kube-prometheus-stack-prometheus-1 in StatefulSet prometheus-kube-prometheus-stack-prometheus successful

You can test this by editing the annotation or environment-variable in the deployment/statefulset and see how it restarts the pods

[Sarab]: Editing the annotation StatefulSet doesn't triggers anything. Probably because the operator may not be programmed to react to changes in annotations.

On the side note, reloader works fine with StatefulSet(s) that are created directly (without using any custom operator) however only causes the trouble with the StatefulSet created/managed by the PrometheusOperator. i.e.

➜  k get statefulset prometheus-kube-prometheus-stack-prometheus -n monitoring -o yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  annotations:
    meta.helm.sh/release-name: kube-prometheus-stack-6e4a97be
    meta.helm.sh/release-namespace: monitoring
    prometheus-operator-input-hash: "4318605777274607378"
    secret.reloader.stakater.com/reload: thanos-sidecar-secure-web
  creationTimestamp: "2024-04-15T02:15:47Z"
  generation: 230
  labels:
    app: kube-prometheus-stack-prometheus
    app.kubernetes.io/instance: kube-prometheus-stack-6e4a97be
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 61.5.0
    chart: kube-prometheus-stack-61.5.0
    managed-by: prometheus-operator
    operator.prometheus.io/mode: server
    operator.prometheus.io/name: kube-prometheus-stack-prometheus
    release: kube-prometheus-stack-6e4a97be
  name: prometheus-kube-prometheus-stack-prometheus
  namespace: monitoring
  ownerReferences:
  - apiVersion: monitoring.coreos.com/v1
    blockOwnerDeletion: true
    controller: true
    kind: Prometheus
    name: kube-prometheus-stack-prometheus
    uid: 5c889be7-d672-45a1-8b00-1b0ef3eb2806
  resourceVersion: "895616441"
  uid: 15a65068-9967-41c5-ad8a-3577e43cff86
  ... REDACTED

[jd-hatzenbuhler]

Hello,

We stumble upon the same issue for the alertmanager sts managed by prometheus operator.
We are using the reloader in reloadStrategy: annotations.
When investigating, we discovered that the prometheus-operator allows only kubectl.kubernetes.io/ annotations to be persisted see https://github.com/prometheus-operator/prometheus-operator/blob/b0d836265833c31b2d730c4edffefda773ee1616/pkg/k8sutil/k8sutil.go#L520
So the reloader will never works.
However in our case we work around the issue by using the native alertmanager reloading mechanism base on secret see https://github.com/prometheus-operator/prometheus-operator/blob/c840b7eaa857fc45f8a9be52cfd8d315f7c88e9f/Documentation/user-guides/alerting.md#using-a-kubernetes-secret

IMO the issue could be closed in reloader side because the issue is more on prometheus-operator controller that doesn't allow annotations to be put by other external controller.