[kube-prometheus-stack] Define custom Probe CRD's within Chart

[meesk]

Is your feature request related to a problem ?

I'm currently writing a Probe definition to scrape some targets using the ingress target (via blackbox exporter), however, there is no way to create a Probe CRD using a built in functionality. Currently I have to write my own template to create these Probe CRD's. As far as I know, this currently is not possible. Some ingresses differ which requires me to write multiple Probe CRD's. It would be nice to create Probes dynamically built into the chart.

If there is already an alternative or any reason why this shouldn't be supported, please let me know.

Describe the solution you'd like.

Provide a built in solution to create Probe CRD's.

Describe alternatives you've considered.

Using my own template, for example:

{{- range .Values.prometheus.prometheusSpec.probes }}
apiVersion: monitoring.coreos.com/v1
kind: Probe
metadata:
  name: {{ .name }}
  namespace: prometheus
  labels:
    release: prometheus
spec:
  jobName: {{ .name }}
  interval: 2m
  prober:
    url: prometheus-blackbox-exporter.prometheus.svc.cluster.local:9115
  targets:
    ingress:
      namespaceSelector:
        matchNames:
          - dev
      selector:
        matchLabels:
        {{- range $key, $val := .matchLabels }}
          {{ $key }}: {{ $val | quote }}
        {{- end }}
      relabelingConfigs:
        - action: replace
          replacement: prometheus-blackbox-exporter.prometheus.svc.cluster.local:9115
          targetLabel: __address__
        - action: replace
          sourceLabels: [__meta_kubernetes_ingress_host, __meta_kubernetes_ingress_path, __meta_kubernetes_ingress_annotation_prometheus_io_probe_path]
          regex: {{ .regex.pattern }}
          replacement: {{ .regex.replacement }}
          targetLabel: __param_target
        - action: replace
          sourceLabels: [__param_target]
          targetLabel: target
        - action: replace
          sourceLabels: [__meta_kubernetes_ingress_name]
          targetLabel: instance
        - action: replace
          sourceLabels: [__meta_kubernetes_ingress_annotation_prometheus_io_module]
          targetLabel: module
        - action: replace
          sourceLabels: [module]
          targetLabel: __param_module
        - action: labeldrop
          regex: __meta_kubernetes_ingress_.*
{{- end }}

Then in my values I currently do something like:

     probes:
        - name: microservice-probe
          matchLabels:
            prometheus.io/probe: 'true' # label defined on ingress
            prometheus.io/probe-template: template-a # label defined on ingress
          regex: 
            pattern: [some-regex-pattern]
            replacement: [some-replacement]

This is ofcourse just an example of my current implementation, it would need more customization.

Additional context.

No response

[jkroepke]

@QuentinBisson @GMartinez-Sisti whats your opinion about the integration of the blackbox_exporter into the kube-prometheus-stack, including Probe CRDs and a default probe which supports simple http?

If agreed, @meesk are you open to contribute the functionality?

[meesk]

@jkroepke yes absolutely. I feel like it should be possible, because it already provides the declaration for the CRD but no way to implement it.

[GMartinez-Sisti]

@QuentinBisson @GMartinez-Sisti whats your opinion about the integration of the blackbox_exporter into the kube-prometheus-stack, including Probe CRDs and a default probe which supports simple http?

I think it makes sense. It's very common to require it with kube-prometheus-stack.

[QuentinBisson]

I agree as well

[jkroepke]

@meesk before going into implementation, I would like to discuss what is possible in context of annotations.

As I know: the following values should be possible to set as annotation on a pod, service, ingress:

prometheus.io/probe: true
prometheus.io/probe-module: http
prometheus.io/probe-port: (similar to scrape port annotation)

Anythink else, what i miss?

I wish, i could offer more, more a potential solution was rejected an a potential solution needs to be designed first.

Ref: prometheus/blackbox_exporter#1098