✨: Add custom Prometheus metrics for controller observability

[MaxRink]

What this PR does

Adds custom Prometheus metrics to all CAPM3 controllers, addressing the lack of visibility into provisioning times, error rates, and controller performance.

Why we need it

Previously, CAPM3 only exposed default controller-runtime metrics, making it difficult to:

• Monitor provisioning success rates and durations
• Debug slow BMH associations
• Track error rates by controller type
• Set up meaningful alerts for operational issues

Changes

New files

• internal/metrics/metrics.go - Central metrics package with 22 Prometheus metrics
• internal/metrics/metrics_test.go - Unit tests for all helper functions (14 tests)
• docs/metrics.md - Comprehensive documentation with PromQL examples

Modified controllers

All controllers instrumented with reconcile metrics:

• metal3machine_controller.go - Also tracks BMH association and provisioning duration
• metal3cluster_controller.go
• metal3data_controller.go
• metal3datatemplate_controller.go
• metal3remediation_controller.go
• metal3machinetemplate_controller.go
• metal3labelsync_controller.go

Metrics overview

Category	Metrics
Reconcile performance	Counter + duration histogram per controller
BMH association	capm3_bmh_association_total, capm3_bmh_association_duration_seconds
Provisioning	capm3_metal3machine_provisioning_duration_seconds (creation → Ready)
Errors	capm3_reconcile_errors_total by controller
Phase tracking	capm3_metal3machine_phase_transitions_total

How to test

make test  # Runs generate + lint + unit tests

[metal3-io-bot]

Hi @MaxRink. Thanks for your PR.

I'm waiting for a metal3-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[tuminoid]

/ok-to-test

I think @fmuyassarov had something like this brewing as well?
/cc @lentzi90 @adilGhaffarDev @kashifest @fmuyassarov

[fmuyassarov]

/ok-to-test

I think @fmuyassarov had something like this brewing as well? /cc @lentzi90 @adilGhaffarDev @kashifest @fmuyassarov

Yes, I was doing some custom metrics addition. But this might be a good start. I will take a look today/tomorrow at this patch.

[fmuyassarov]

Can you rebase before we start reviewing?

[fmuyassarov]

ping @MaxRink

[Copilot]

Pull request overview

Adds a centralized custom Prometheus metrics package and instruments CAPM3 controllers to improve observability into reconcile performance, provisioning workflows, and error rates.

Changes:

• Introduces internal/metrics with CAPM3-specific counters/histograms/gauges and helper recording functions.
• Instruments multiple controllers to record reconcile duration/result metrics, plus additional Metal3Machine provisioning and BMH association metrics.
• Adds metrics documentation (docs/metrics.md) and basic unit tests for metric helper functions.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 14 comments.

Show a summary per file

File	Description
internal/metrics/metrics.go	Defines and registers CAPM3 Prometheus metrics plus helper recording functions.
internal/metrics/metrics_test.go	Adds unit tests covering metric helper entrypoints.
docs/metrics.md	Documents exposed metrics, labels, and example PromQL queries.
go.mod	Promotes prometheus/client_golang to a direct dependency.
controllers/metal3machine_controller.go	Adds reconcile + BMH association + provisioning duration instrumentation.
controllers/metal3machine_controller_test.go	Updates mocks/calls to accommodate additional provisioning/metrics logic.
controllers/metal3cluster_controller.go	Adds reconcile + error instrumentation.
controllers/metal3data_controller.go	Adds reconcile + error instrumentation.
controllers/metal3datatemplate_controller.go	Adds reconcile + error instrumentation.
controllers/metal3remediation_controller.go	Adds reconcile + error instrumentation.
controllers/metal3machinetemplate_controller.go	Adds reconcile + error instrumentation.
controllers/metal3labelsync_controller.go	Adds reconcile instrumentation.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Sunnatillo]

@fmuyassarov needs your review

[tuminoid]

/cc @fmuyassarov @lentzi90

[adilGhaffarDev]

/approved
thanks @MaxRink
One minor comment but not a blocker.
/hold
holding so others can review.

[metal3-io-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: adilGhaffarDev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [adilGhaffarDev]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[adilGhaffarDev]

looks good.

[lentzi90]

/test metal3-centos-e2e-integration-test-main metal3-ubuntu-e2e-integration-test-main

[lentzi90]

Looking good overall! I have a couple of comments on passing through the function to record reconciliation errors. This I would like to get rid of so we just do it with the defer. What do you think about this?

[lentzi90]

Could you please squash the commits?

[lentzi90]

/test metal3-ubuntu-e2e-integration-test-main metal3-centos-e2e-integration-test-main

[lentzi90]

/lgtm

[lentzi90]

nit: 2026

[lentzi90]

nit: 2026

[metal3-io-bot]

New changes are detected. LGTM label has been removed.