Problems with Grafana Dashboards

[jseiser]

We have a kube-prometheus-stack running, with 1 Cluster Wide install of the Operator, and 1 3 node cluster running.

The Operator

    replicaCount: 2
    watchNamespaces: ["mimir"]
    podDisruptionBudget:
      enabled: true
    dashboards:
      enabled: true
      namespace: ~
      label: grafana_dashboard # this is the default value from the grafana chart
      labelValue: "1" # this is the default value from the grafana chart
      annotations: {}
      extraLabels:
        app.kubernetes.io/part-of: strimzi-kafka-operator
        app.kubernetes.io/component: monitoring
        release: kube-prometheus-stack

These dashboards all show up in Grafana, so this portion is working.

We have the kube-state-metrics stuff patched in: https://gist.github.com/jseiser/5b6e4a01e51cbb3e3d76e99d701f0edb

This appears to work, I can see metrics from it, in prometheus

strimzi_pod_set_resource_info{cluster="eks-prod-cluster", container="kube-state-metrics", currentPods="3", customresource_group="core.strimzi.io", customresource_kind="StrimziPodSet", customresource_version="v1", endpoint="http", exported_namespace="mimir", generation="1", instance="10.16.19.61:8080", job="kube-state-metrics", name="mimir-mimir", namespace="monitoring", pod="kube-prometheus-stack-kube-state-metrics-5b65b7c4fc-zsh27", pods="3", readyPods="3", service="kube-prometheus-stack-kube-state-metrics"}

Relevant parts of my kind: Kafka

apiVersion: kafka.strimzi.io/v1
kind: Kafka
metadata:
  name: mimir
spec:
  kafka:
    metricsConfig:
      type: strimziMetricsReporter
  kafkaExporter:
    topicRegex: ".*"
    groupRegex: ".*"

I then created a podMonitor

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: mimir-kafka-metrics
  namespace: mimir
  labels:
    name: mimir-kafka-metrics
    release: kube-prometheus-stack
spec:
  selector:
    matchLabels:
      strimzi.io/name: mimir-kafka
  namespaceSelector:
    matchNames:
      - mimir
  podMetricsEndpoints:
    - path: /metrics
      port: tcp-prometheus

The end result is.. I have some stuff, but most of the dashboards have nothing. I understand that Strimzi Kafka Bridge will be empty, as we are not running any Kafka Bridges. The Strimzi Kafka dashboard shows Brokers Online, Active Controllers and Online Replicas, nothing else.

When you look at the variables, a good amount are not returning anything.

Example would be when it looks for the Strimzi Cluster Name

kafka_server_replicamanager_leadercount{namespace="$kubernetes_namespace"}

It has a regex

/.*strimzi_io_cluster="([^"]*).*/

Nothing returned by that metric, matches the regex.

kafka_server_replicamanager_leadercount{cluster="eks-cluster", container="kafka", endpoint="tcp-prometheus", instance="10.16.23.124:9404", job="mimir/mimir-kafka-metrics", namespace="mimir", pod="mimir-mimir-1"}

[scholzj]

strimzi_io_cluster would likely be taken from the label on the Kubernetes Pod it was scraped from. That would likely be done by this rule I guess? https://github.com/strimzi/strimzi-kafka-operator/blob/main/examples/metrics/prometheus-install/pod-monitors/kafka-resources-metrics.yaml#L21-L24 ... so you might need to check why are these not applied in your setup.

[jseiser]

thanks, I missed everything that wasnt under strimzi-metrics-reporter