How to monitor a K3s cluster using Grafana?

[shangjianan2]

I want to monitor a K3s cluster using Grafana + Prometheus. Currently, Prometheus can collect K3s cluster information normally, and Grafana is also connected to Prometheus normally. However, several dashboards loaded from "https://grafana.com/grafana/dashboards/" cannot display data properly. I checked the dashboard configurations and found that the metrics specified in the dashboards do not match those in the actual K3s cluster. How can I solve this problem?

k3s version v1.32.3+k3s1 (079ffa8)
go version go1.23.6

The environment is offline and cannot connect to the Internet.

The Grafana resource manifest is as follows.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-config-pv
spec:
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  nfs:
    server: 10.111.160.26
    path: "/data/nfs/monitoring/grafana/data/etc/grafana"
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: grafana-data-pv
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteMany
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  nfs:
    server: 10.111.160.26
    path: "/data/nfs/monitoring/grafana/data/var/lib/grafana"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-config-pvc
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 1Gi
  storageClassName: ""
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-data-pvc
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 5Gi
  storageClassName: ""
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      nodeSelector:
        zbwt-node-name: node3
      securityContext:
        runAsUser: 0
        runAsGroup: 0
      hostAliases:
        - ip: "10.111.160.21"  # SMTP 服务器 IP
          hostnames:
            - "smtp.163.com"  # 为 SMTP 服务器添加域名解析
      containers:
        - name: grafana
          image: 10.111.160.21:39435/grafana/grafana-enterprise:latest
          ports:
            - containerPort: 3000
          env:
            - name: GF_SECURITY_ADMIN_USER
              value: "admin"
            - name: GF_SECURITY_ADMIN_PASSWORD
              value: "ZBbdtech@4"
          volumeMounts:
            - name: grafana-config
              mountPath: /etc/grafana/grafana.ini
              subPath: grafana.ini
            - name: grafana-data
              mountPath: /var/lib/grafana
      volumes:
        - name: grafana-config
          persistentVolumeClaim:
            claimName: grafana-config-pvc
        - name: grafana-data
          persistentVolumeClaim:
            claimName: grafana-data-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  type: NodePort
  selector:
    app: grafana
  ports:
    - name: http
      port: 3000
      targetPort: 3000
      nodePort: 30012

The Prometheus resource manifest is as follows.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - nodes/metrics
      - nodes/proxy
      - services
      - endpoints
      - pods
    verbs: ["get", "list", "watch"]
  - apiGroups: ["extensions"]
    resources:
      - ingresses
    verbs: ["get", "list", "watch"]
  - nonResourceURLs: ["/metrics", "/healthz"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
  - kind: ServiceAccount
    name: prometheus
    namespace: monitoring

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s

    scrape_configs:
      - job_name: 'kubernetes-nodes'
        kubernetes_sd_configs:
          - role: node
        scheme: https
        tls_config:
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        metrics_path: /metrics
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics

      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
          - role: pod
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
            action: keep
            regex: true
          - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
            action: replace
            target_label: __metrics_path__
            regex: (.+)
          - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
            action: replace
            regex: (.+):(?:\d+);(\d+)
            replacement: ${1}:${2}
            target_label: __address__

      - job_name: 'node1-occupation-transfer'
        metrics_path: '/actuator/prometheus'
        static_configs:
          - targets: ['cicd-occupation-transfer-svc-1.zbwt-prod.svc.cluster.local:8085']

      - job_name: 'node2-occupation-transfer'
        metrics_path: '/actuator/prometheus'
        static_configs:
          - targets: ['cicd-occupation-transfer-svc-2.zbwt-prod.svc.cluster.local:8085']

      - job_name: 'node3-occupation-transfer'
        metrics_path: '/actuator/prometheus'
        static_configs:
          - targets: ['cicd-occupation-transfer-svc-3.zbwt-prod.svc.cluster.local:8085']

      - job_name: 'node4-occupation-transfer'
        metrics_path: '/actuator/prometheus'
        static_configs:
          - targets: ['cicd-occupation-transfer-svc-4.zbwt-prod.svc.cluster.local:8085']

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      nodeSelector:
        zbwt-node-name: node3
      containers:
        - name: prometheus
          image: 10.111.160.21:39435/ubuntu/prometheus:latest
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
            - "--storage.tsdb.retention.time=15d"
            - "--storage.tsdb.retention.size=30GB"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: config
              mountPath: /etc/prometheus
            - name: storage
              mountPath: /prometheus
      volumes:
        - name: config
          configMap:
            name: prometheus-config
        - name: storage
          nfs:
            server: 10.111.160.26
            path: /data/nfs/monitoring/prometheus/data
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
  namespace: monitoring
spec:
  type: NodePort
  ports:
    - port: 9090
      targetPort: 9090
      nodePort: 30011
  selector:
    app: prometheus

prometheus.yml

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'kubernetes-nodes'
    kubernetes_sd_configs:
      - role: node
    scheme: https
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    metrics_path: /metrics
    relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - target_label: __address__
        replacement: kubernetes.default.svc:443
      - source_labels: [__meta_kubernetes_node_name]
        regex: (.+)
        target_label: __metrics_path__
        replacement: /api/v1/nodes/${1}/proxy/metrics

  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (.+):(?:\d+);(\d+)
        replacement: ${1}:${2}
        target_label: __address__

  - job_name: 'node1-occupation-transfer'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['cicd-occupation-transfer-svc-1.zbwt-prod.svc.cluster.local:8085']

  - job_name: 'node2-occupation-transfer'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['cicd-occupation-transfer-svc-2.zbwt-prod.svc.cluster.local:8085']

  - job_name: 'node3-occupation-transfer'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['cicd-occupation-transfer-svc-3.zbwt-prod.svc.cluster.local:8085']

  - job_name: 'node4-occupation-transfer'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['cicd-occupation-transfer-svc-4.zbwt-prod.svc.cluster.local:8085']

I believe that Prometheus has successfully connected to the K3s cluster and collected monitoring metrics through the following screenshots.

I believe Grafana is properly connected to Prometheus through the following screenshots.

I used the configuration from the link "https://grafana.com/grafana/dashboards/16450-kubernetes-views-k3s-cluster/", but it didn't display any data.

I checked the configuration file provided in this link and found that one of the metrics is node_cpu_seconds_total, but this metric does not exist in the K3s cluster.

I've tested other K3s-related dashboard configurations from "https://grafana.com/grafana/dashboards/" following the steps above, but none of them work. Could there be an issue with my setup that's causing these official Grafana dashboards to fail?