enable and scrape CSI driver metrics

[matthias-horne]

How to categorize this PR?

/area monitoring
/kind enhancement
/platform aws

What this PR does / why we need it:

This PR enables metrics for the CSI driver and adds a ServiceMonitor to scrape them.

Which issue(s) this PR fixes:
Fixes #1568

Special notes for your reviewer:

Release note:

Metrics for the CSI driver controller are now enabled

[github-actions]

This change adds comprehensive monitoring capabilities to the CSI driver controller by exposing metrics on port 3301 and integrating with Prometheus for observability of AWS EBS CSI operations.

Walkthrough

• New Feature: Added metrics endpoint exposure on port 3301 to the CSI driver controller for monitoring and observability
• New Feature: Created Kubernetes Service to expose the metrics port with proper networking annotations for scraping access
• New Feature: Implemented ServiceMonitor configuration for Prometheus integration, enabling collection of AWS EBS CSI-specific metrics including API request duration, errors, throttles, and EC2 detach pending time

_{Model: claude-sonnet-4-20250514 | Prompt Tokens: 922 | Completion Tokens: 143}

[hebelsan]

In the issue, you mentioned that we should consider putting this feature behind a flag so it can be enabled on demand, given that it could scale into an unbounded number of metrics.
What led to the decision to enable it by default now?

[matthias-horne]

In the issue, you mentioned that we should consider putting this feature behind a flag so it can be enabled on demand, given that it could scale into an unbounded number of metrics. What led to the decision to enable it by default now?

I did take a detailed look at the exposed metrics and discussed with @rickardsjp. The following table shows the number of individual metrics that will be generated.

Metric	Calculation	Count
aws_ebs_csi_api_request_duration_seconds	16 request types * 14 metrics/type	224
aws_ebs_csi_api_request_errors_total	16 request types * 1 counter	max 16
aws_ebs_csi_api_request_throttles_total	16 request types * 1 counter	max 16
aws_ebs_csi_ec2_detach_pending_seconds	1 counter * number of volumes pending detachment 1	max number of PVC

We concluded that the number of additional metrics should not be too much for Prometheus. Worst case the retention time might be reduced. The low risk combined with a need to timely deliver the additional metrics justifies delivering this feature without an additional flag.

Footnotes

1. only created when volume is not detached in the first attempt. Deleted as soon as volume is successfully detached. If this happens between two scrapes, the metric might never show up in Prometheus. ↩

[hebelsan]

/lgtm
/approve

[gardener-prow]

LGTM label has been added.

Details

Git tree hash: a71ed1a188338237de34ef4f3397101d7bb8f6e4

[gardener-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: hebelsan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [hebelsan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment