[Feature]: Prometheus Metrics Abstraction

[mladjan-gadzic]

🚀 The feature, motivation and pitch

There is a need for OpenTelemetry metrics support.

Just adding OTel metrics support on top of current state of the project is going to increase maintenance effort. Any metric added to OTel needs to be added to Prometheus and vise-versa. To simplify it - instead of adding metric to a single place, it needs to be added to two places.

Because of above reasoning, the idea is to create metrics abstraction so that metrics are maintainable. To change or add new metrics - modification is needed only in one place. User can decide further whether he'd like to use Prometheus based metrics (default) or opt for OTel metrics or both.

Alternatives

No response

Additional context

For more context check this PR #30258

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[markmc]

Note - I would be open to use switching completely to the OpenTelemetry library if it could export metrics in a Prometheus-compatible format, allowing us to switch libraries without breaking any users. After a little research, it doesn't seem like that compatibility is trivial - https://opentelemetry.io/blog/2024/prometheus-compatibility-survey/

Also, our use of prometheus_client multiprocess_mode for the handful of gauges we have is important when we have multiple API servers (--api-server-count > 1) so it's important we consider whether OpenTelemetry can implement this

[punitmahes]

@mladjan-gadzic Good Initiative. I wanted to check if there are any talks about moving to common Semantic Conventions for all these metrics Since It goes different with different inference framework. This abstraction framework can also handle the SemConv related conversions if both metrics need to be supported. What are your thoughts?

[mladjan-gadzic]

Note - I would be open to use switching completely to the OpenTelemetry library if it could export metrics in a Prometheus-compatible format, allowing us to switch libraries without breaking any users. After a little research, it doesn't seem like that compatibility is trivial - https://opentelemetry.io/blog/2024/prometheus-compatibility-survey/

Also, our use of prometheus_client multiprocess_mode for the handful of gauges we have is important when we have multiple API servers (--api-server-count > 1) so it's important we consider whether OpenTelemetry can implement this

this is something that needs to be taken into consideration when otel metrics are to be implemented. i agree.

this issue is more about abstraction of metrics itself so those can be prepared for incoming otel support. i am gonna put link to this comment on otel support issue.

[mladjan-gadzic]

@mladjan-gadzic Good Initiative. I wanted to check if there are any talks about moving to common Semantic Conventions for all these metrics Since It goes different with different inference framework. This abstraction framework can also handle the SemConv related conversions if both metrics need to be supported. What are your thoughts?

thanks! i am not aware if there are any talks. however, i agree that having semconv conversion is a nice touch! the beauty of abstraction of these metrics is that semconv translation can be added aftwerwards as an opt-in.

[markmc]

Note - I would be open to use switching completely to the OpenTelemetry library if it could export metrics in a Prometheus-compatible format, allowing us to switch libraries without breaking any users. After a little research, it doesn't seem like that compatibility is trivial - https://opentelemetry.io/blog/2024/prometheus-compatibility-survey/

[..]

this issue is more about abstraction of metrics itself so those can be prepared for incoming otel support.

Just to be clear - my main point is that if the OpenTelemetry library could compatibly expose all metrics in Prometheus support, there would be no need for an abstraction - the OpenTelemetry library would be that abstraction. (I'm not hopeful that this is possible, but one can dream!)

[mladjan-gadzic]

created a draft pr for this #30689. looking for an early feedback. please let me know your opinion.