Prometheus Metrics

[stefanhipfel]

Summary

Add Prometheus Metrics for BMC/Server

Basic example

Could be added via the BMC crd:

apiVersion: metal.ironcore.dev/v1alpha1
kind: BMC
metadata:
  labels:
    app.kubernetes.io/name: bmc
    app.kubernetes.io/instance: bmc-sample
    app.kubernetes.io/part-of: metal-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: metal-operator
  name: bmc-sample
spec:
  endpointRef:
    name: endpoint-sample
  bmcSecretRef:
    name: bmc-sample
  protocol:
    name: Redfish
    port: 8080
  prometheusMetrics:
    scrape: true
    path: '/data/metrics'
    port: '80'
    image: 'exporter:latest'
    

The BMC controller would create a Pod with the image and annotations provided in the spec.

Motivation

It is important to monitor bmc/server sensor information and alert on possible failures

[afritzler]

Why don't we expose additional metrics via the manager and hook into the already existing setup: https://book.kubebuilder.io/reference/metrics#publishing-additional-metrics

[afritzler]

Alternatively we could think about how we can leverage existing BMC exporters which are out there e.g. https://github.com/gebn/bmc_exporter

Still we would need to add support for our crds for a seamless integration.
Eric however made me aware that our crds are running in a workerless cluster. So the manager cannot create any pods or jobs atm.

[stefanhipfel]

Why don't we expose additional metrics via the manager and hook into the already existing setup: https://book.kubebuilder.io/reference/metrics#publishing-additional-metrics

Ok but then one manager would need to fetch metrics from each node? The time to fetch and amount of data for all nodes is a lot ;)
Otherwise we would need a discovery service.

[afritzler]

Another advantage of starting a routine in the manager is that you can pass on the client via which you can retrieve all the BMCs and Servers in the system.

[stefanhipfel]

Putting the metrics exporter in a pod next to the manager, allows for easy scaling (50/50 server handling etc). Also it won’t affect the manager in case of long running requests and huge metrics data which causes high cpu and memory usage.

This setup could use the same client config as well.

For a small amount of servers, I agree using a goroutine in the manager. For >1k servers I have my doubt, that it scales well.

We could start developing an exporter go binary which should run in the managers goroutine or a pod.

[stefanhipfel]

Dell Telemetric -> Prometheus: https://github.com/dell/iDRAC-Telemetry-Reference-Tools