[kube-prometheus-stack] - no default dashboard or data source

[fjellvannet]

Describe the bug a clear and concise description of what the bug is.

I installed kube-prometheus-stack into the monitoring-namespace in my bare-metal kubeadm-cluster.

However, when I log into grafana, there are no dashboards visible, and prometheus also does not appear as the default datasource.

What's your helm version?

version.BuildInfo{Version:"v3.12.3", GitCommit:"3a31588ad33fe3b89af5a2a54ee1d25bfe6eaa5e", GitTreeState:"clean", GoVersion:"go1.20.7"}

What's your kubectl version?

Client Version: v1.32.1 Kustomize Version: v5.5.0 Server Version: v1.32.1

Which chart?

prometheus-community/kube-prometheus-stack

What's the chart version?

68.3.0

What happened?

I tried to install the kube-prometheus-stack using the following command:
helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

All the pods / services etc are deployed properly, and I can log into grafana using a port-forward or ingress.

However - there are no dashboards in grafana, and also there is no datasource (prometheus is not automatically added).

What you expected to happen?

I would expect the default dashboards for the different kubernetes components like etcd etc. to be visible and populated with data. Instead, Grafana appears like a clean instance with nothing configured.

How to reproduce it?

Set up a bare-metal kubeadm cluster on ubuntu nodes, I now have 3 workers and 3 masters. For block- and other persistent storage, rook-ceph is configured on the worker nodes. Additionally, serverTLSBootstrap is enabled in the kubelets as described here for metrics-server to work. Cert-manager is also running in the cluster. Other than that it is pretty bare-metal, and I have no clue why I don't see this data.

Enter the changed values of values.yaml?

NONE

I tried enabling persistent storage for grafana, enabled an ingress for grafana, set the url of the prometheus-datasource in grafana to this value: http://kube-prometheus-stack-prometheus.monitoring.svc.cluster.local:9090/, but it all made no difference. I see no dashboards and no datasource, even though they are all true in the values. While testing I have also not only helm upgraded, but also helm uninstall and then again helm install to be sure that everything is reconfigured.

I purposely created the issue with NONE as the modified values, because I think it would make sense have the default values work more or less out of the box.

Enter the command that you execute and failing/misfunctioning.

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring --create-namespace

Anything else we need to know?

I have tried to follow the different hints I found in the related issues #873 and #3800. See changed values of values.yaml to see what I tried to modify. As both of these issues were old and opened and closed multiple times, I decided to open a new clean issue to get this fixed.

[chris8205]

Same issue...
also recognized that default DataSources --> Prometheus and Alermanager not registred properly...
Also Prune not helping...

also "version: x" not added in default "datasources.yaml"

apiVersion: v1
data:
  datasource.yaml: |-
    apiVersion: 1
    prune: true
    datasources:
    - name: "Prometheus"
      type: prometheus
      uid: prometheus
      url: http://observability-prometheus.<mynamespace>:9090/
      access: proxy
      isDefault: true
      jsonData:
        httpMethod: POST
        timeInterval: 1m
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: xxxxxxxxxxxxxxx
    meta.helm.sh/release-namespace: xxxxxxxxxxxxxxxx
  creationTimestamp: "2025-01-27T10:46:16Z"
  labels:
    app: kube-prometheus-stack-grafana
    app.kubernetes.io/instance: xxxxxxxxxxxxxxxx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/part-of: kube-prometheus-stack
    app.kubernetes.io/version: 68.1.1
    chart: kube-prometheus-stack-68.1.1
    grafana_datasource: "1"
    heritage: Helm
    release: xxxxxxxxxxxxxxxxxx
  name: observability-grafana-datasource
  namespace: xxxxxxxxxxxxxxxx
  resourceVersion: "xxxxxxxxxxx"
  uid: xxxxxxxx-xxxx-xxxx-xxxxx-xxxxxxxxxxxx

[audiophonicz]

Same issue...
I just went to install a clean install of kube-prometheus-stack 70.4.1 from 4/1/25 and everything seems to be running properly except I have no dashboards or datasources in grafana. I also have added persistent storage to both prometheus and grafana.

3master x 6worker NUC cluster with K3s 1.30.6 on Debian 12.7 Kernel 6.10
Helm v3.11.2

Edit: it looks like the whole ConfigMap: prometheus-grafana-datasource is not even listed as a volume in the grafana deployment let alone mounting the subpath "datasource.yaml" to emptyDir "prometheus-grafana-datasource" even though the sidecar datasources are all enabled by default.