Support BMC monitoring without relying on event subscriptions

[stefanhipfel]

Problem

The metal-operator currently relies on Redfish event subscriptions for BMC monitoring. However, many BMC vendors/models do not support event subscriptions reliably:

• HPE iLO has limited DeliveryRetryPolicy support
• Older BMC firmware versions lack EventService entirely
• Some vendors have incomplete or buggy event implementations
• Event subscriptions can be lost during BMC resets/firmware updates

This limits our ability to monitor BMCs consistently across heterogeneous environments.

Solution

Implement a polling-based monitoring system as the default monitoring mode with flexible deployment options.

Key Features

• Universal compatibility: Polling works with all BMC vendors (no EventService required)
• Flexible deployment: Run monitoring as part of the operator (default) OR as a standalone pod
• Smart caching: Collected metrics stored in memory; Prometheus scrapes from cache (not directly from BMCs)
• Efficient: Session pooling prevents exhausting BMC connection limits
• Vendor-aware: Handles endpoint variations across HPE, Dell, Supermicro, Lenovo
• HA-ready: Leader election ensures only one instance collects metrics

Architecture Overview

Two Deployment Options:

1. Embedded Mode (Default): Monitoring runs as a component within the operator pod

   • Simpler deployment
   • Shared resources with operator
   • Good for most use cases

2. Standalone Mode: Monitoring runs as a separate deployment

   • Dedicated resources for monitoring
   • Independent scaling
   • Better for large deployments (>500 BMCs)
   • Can be updated independently from operator

Data Flow:

BMCs → Polling/Events → In-Memory Cache → Prometheus

The cache is key: it decouples Prometheus scraping (every 15-30s) from BMC polling (every > 120s), preventing excessive load on BMCs.

What Gets Monitored

• Sensors: Temperature, fan speeds, power supply status, voltages
• Alerts: Critical events from BMC event logs
• SEL: System Event Log entries for hardware events

Monitoring Modes

• Polling (default): Actively queries BMC endpoints every 60-120 seconds

  • Works with all BMCs
  • Predictable load pattern

• Events: Uses Redfish event subscriptions (existing implementation)

  • Lower latency when it works
  • Only for BMCs with reliable EventService support

Mode is configured globally for all BMCs via operator CLI flag.

Implementation Plan

Phase 1: Foundation

• Design architecture and interfaces
• Implement embedded monitoring runnable
• Implement standalone deployment option
• Add configuration and validation

Phase 2: Data Collection

• Sensor polling (Thermal, Power, Sensors endpoints)
• Alert/event log polling
• SEL polling

Phase 3: Integration

• BMC status updates
• Prometheus alerts and Grafana dashboards

Phase 4: Quality

• Unit tests (≥80% coverage)
• Integration tests with Redfish simulator
• Documentation

Acceptance Criteria

• Polling works with all BMC vendors (HPE, Dell, Supermicro, Lenovo)
• Can deploy as embedded runnable or standalone pod
• Metrics cached in memory (Prometheus scrapes don't hit BMCs)
• Session pooling prevents BMC connection exhaustion
• Sensor, alert, and SEL data collected and exposed via Prometheus
• BMC status reflects monitoring health
• Works in HA deployments (leader election)
• Comprehensive tests and documentation

Sub-Issues

Will be created to break down implementation into manageable tasks across 4 phases.

Non-Goals

• Historical data storage (use Prometheus for this)
• BMC firmware updates or auto-remediation
• Custom alerting (users configure Prometheus/Alertmanager)
• Multi-cluster monitoring

References

• Current event subscription: internal/serverevents/
• BMC controller: internal/controller/bmc_controller.go
• Redfish client: bmc/redfish.go

[coderabbitai]

⚠️ Possible Duplicate Issue(s)

• Prometheus Metrics #474

🔗 Related PRs

#579 - Improve Dell hardware support with enhanced firmware updates and BMC configuration [merged]
#718 - Add Prometheus metrics and alerts for server monitoring [merged]
#733 - Implement vendor-specific BMC/BIOS upgrade checks with firmware inventory function [merged]
#754 - Fix BMC event subscription errors on production hardware [merged]
#756 - Fix/handle duplicate subscriptions [merged]

---

📝 Issue Planner

_{Check the box below or use the @coderabbitai plan command to generate an implementation plan and prompts that you can use with your favorite coding assistant.}

• Create Plan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[maxmoehl]

I would kindly ask you to re-phrase this in a way that it can be processed by a human. The focus of an issue should be on the issue / requirement we want to address, the title shows that this issue is not centered on a problem but rather on a solution:

Implement Polling-Based BMC Monitoring for Sensors, Alerts, and SEL

Issues should rather be phrased like this:

Support BMC monitoring without relying on event subscriptions

The first section covers the problem statement well but what follows is irrelevant to the discussion of how to resolve the problem. The goal is to have a rough sketch of possible implementations instead of basically the entire implementation as code blocks.

[maxmoehl]

Thank you for updating the issue, I still don't like that it's clearly written by an LLM with a focus on an implementation plan but at least the size is manageable now.

This limits our ability to monitor BMCs consistently across heterogeneous environments.

I'm assuming "heterogeneous environments" refers to a single installation with mixed vendors? In such an environment with a global flag the user has then to decide whether to use the less efficient method to get all data or the more efficient one but missing out on data for vendors that don't support it. In homogeneous environments this isn't an issue, but for heterogenous ones I would question whether a global flag is the right solution.

• Sensors: Temperature, fan speeds, power supply status, voltages
• Alerts: Critical events from BMC event logs
• SEL: System Event Log entries for hardware events

You list these types of data to monitor but to me it seems like the sensors are metrics, but the alerts and systems event log are rather log statements to emit. How would these be reported given that prometheus is focused around metrics and not logs?

[stefanhipfel]

Yes we have regions with a mix of vendors and models. I was expecting, that a BMC would show if Event Subscription and Metrics Report is supported. However for 2 of our Lenovos, subscriptions were successfully created, but no data was sent. So this might be a bug or Lenovo never added this feature completely for those models. We are in contact with Lenovo to figure out what the issue is. However, this makes it hard to automatically fall back to polling sensor data.
So we need to make it configurable: globally, individually per server or via a server/bmc label selector.
Andreas also mentioned that maybe we should just support one option for now.

You list these types of data to monitor but to me it seems like the sensors are metrics, but the alerts and systems event log are rather log statements to emit. How would these be reported given that prometheus is focused around metrics and not logs?

We would just read the number of Critical events and export those numbers to Prometheus and alert for example when increase(number_critical_events) > 0.
This metrics was requested by another team, since they currently read this directly from the vendor consoles.

[afritzler]

Given the experience with the Lenovo BMCs where subscriptions were created successfully but never actually delivered any data, I think we should just go with polling first and not bother with the event subscription mechanism for now.

The main issue is that even when subscriptions "work" (no error on creation), there's no good way to tell they're silently failing. Building automatic fallback logic around that seems like a lot of complexity for something we can't reliably detect anyway.

Polling at 60-120s with session pooling and caching should be perfectly fine for our use case. We can always revisit event subscriptions later if there's a concrete need for lower latency, but right now it just adds another code path to maintain across vendors that may or may not actually work

[afritzler]

For the implementation, I'd take the existing internal/serverevents/server.go and evolve it into something like a metrics server. Right now it's a passive HTTP server that waits for BMCs to push events and metric reports to it. We'd turn that around so it actively polls the BMC Redfish endpoints for sensor data, alerts, and SEL entries, stores everything in the collector cache like it already does, and exposes that to Prometheus. It already implements Start() so it works as a manager runnable. We just keep that and swap the internals from "listen for incoming pushes" to "poll on a ticker". Then we can drop the whole event subscription creation logic and simplify things quite a bit. Wdyt?

[maxmoehl]

For that to work we also need to include some logic to push the information on which endpoints to poll to the runnable. There could be some shared data structure protected by a lock which the BMC reconciler uses to inform the poller of which endpoints to poll.

The details of which we can discuss on the PR I'd say.