[grafana] After 10.2.0 helm chart upgrade datasources sidecar get stuck with skipReload option

[pozniakas]

Due to this issue: #981, we need to set these parameters:

datasources:
  enabled: true
  skipReload: true # workaround due to bug with reloading failing auth if admin.existingSecret is not set: https://github.com/grafana/helm-charts/issues/981
  initDatasources: true # workaround due to bug with reloading failing auth if admin.existingSecret is not set: https://github.com/grafana/helm-charts/issues/981

And with these options and 10.2.0 Helm chart version grafana-init-sc-datasources sidecar gets stuck, but no errors in logs could be found:

{"time": "2025-11-24T15:59:50.833704+00:00", "level": "INFO", "msg": "Starting collector"}
{"time": "2025-11-24T15:59:50.833925+00:00", "level": "INFO", "msg": "No folder annotation was provided, defaulting to k8s-sidecar-target-directory"}
{"time": "2025-11-24T15:59:50.834085+00:00", "level": "INFO", "msg": "Loading incluster config..."}
{"time": "2025-11-24T15:59:50.834729+00:00", "level": "INFO", "msg": "Config for cluster api at 'https://10.96.0.1:443' loaded."}
{"time": "2025-11-24T15:59:50.834838+00:00", "level": "INFO", "msg": "Unique filenames will not be enforced."}
{"time": "2025-11-24T15:59:50.834917+00:00", "level": "INFO", "msg": "5xx response content will not be enabled."}
{"time": "2025-11-24T15:59:50.835178+00:00", "level": "INFO", "msg": "Performing list-based sync on secret resources: {}"}
{"time": "2025-11-24T15:59:50.845177+00:00", "level": "INFO", "msg": "Performing list-based sync on configmap resources: {}"}
{"time": "2025-11-24T15:59:50.859312+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/production-loki-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.860163+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/production-frontend-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.860848+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/production-loki-pgaudit-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.861650+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/production-prometheus-datasource.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.862281+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/production-tempo-datasource.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.862925+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/staging-loki-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.863808+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/staging-frontend-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.864508+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/staging-loki-pgaudit-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.865309+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/staging-prometheus-datasource.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.865923+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/staging-tempo-datasource.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.866521+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/uat-loki-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.867128+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/uat-frontend-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.867703+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/uat-loki-pgaudit-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.868380+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/uat-prometheus-datasource.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.869077+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/uat-tempo-datasource.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.869716+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/loki-scalable-ds.yaml (ascii)"}
{"time": "2025-11-24T15:59:50.870314+00:00", "level": "INFO", "msg": "Writing /etc/grafana/provisioning/datasources/datasource.yaml (ascii)"}

The container just stuck in Running state and nothing happens. But if I revert one version back to 10.1.5 everything starts working and interesting is that the logs of sidecar looks identical. So hard to tell what is the main reason for this bug :)

[carlosmt86]

Same issue here, I need to define the datasources.watchMethod to mitigate the issue.

     datasources:
        watchMethod: "LIST"

this setting was changed on helm 10.2.0 version from harcoded "LIST" to default helm chart value "WATCH":
https://github.com/grafana/helm-charts/pull/4011/files#diff-23be7f77a732d6c6ecd4d8dfa6e944481ee2f7631d9061ce5aa128cb4b2ff410L211-R217

[pozniakas]

Same issue here, I need to define the datasources.watchMethod to mitigate the issue.

     datasources:
        watchMethod: "LIST"

this setting was changed on helm 10.2.0 version from harcoded "LIST" to default helm chart value "WATCH": https://github.com/grafana/helm-charts/pull/4011/files#diff-23be7f77a732d6c6ecd4d8dfa6e944481ee2f7631d9061ce5aa128cb4b2ff410L211-R217

Thanks a lot! After specifying this variable everything seems to work just fine:)

[phoenix-frozen]

This actually didn't work for me, I had to add: (using kube-prometheus-stack, which embeds the grafana chart)

grafana:
  sidecar:
    datasources:
      watchMethod: "SLEEP"

At a guess, this means the new thing, WATCH, is very very broken. (Also seemingly SLEEP replaced LIST, and I suspect that entirely breaking LIST was also not intended.)