Dynamic labels for go-grpc-prometheus middleware

[jzelinskie]

I hope this is the right place for this issue, as I know the repositories are in-flight.

We're implementing customer-facing metrics, so we'd like to be able to label requests with the customer associated with the request.

My naive suggestion is that the middleware take a callback for calculating additional labels.

@bwplotka recommended a workaround that might be possible: using wrapped registries with the middleware.

[johanbrandhorst]

I'm not sure I follow exactly, do you have an example or POC PR? I think we're still pretty open to changing things since v2 isn't finalized yet.

[jzelinskie]

The idea would be to have an upstream-blessed way to add additional labels (that may have to be computed) to the metrics exposed by the middleware.

For example, we wanted to add a label "tenant" to the histogram labels so that we could show our users performance graphs of their own requests against our service.

Our solution is that we use this middleware without the histogram metrics, then we implemented our own histogram with the same name and labels (plus tenant) and a middleware which times the request and populates the tenant label with a value from the context.Context.

[benjibuiltit]

Is there any update on this issue?

I've worked w/ Prometheus and accompanying metric middlewares for HTTP servers across multiple languages and this is the first time I've encountered a lack of support for runtime labeling.

I have a use case very similar to what @jzelinskie has described.

Would you all be open to reviewing a PR for support of a callback with a context.Context argument for the current request?

[johanbrandhorst]

No news here so far, but we have shipped v2 now, so if it requires breaking changes we'd have to consider it carefully. Definitely open to reviewing a PR, though I'm not sure I understand what you're getting at still. Feel free to open on though and we can discuss :).

[benjibuiltit]

It doesn't seem as though there is a way to add a label to metrics using request context, which IIUC is what this issue was created about.

I know that there is support for exemplars, and there is the exemplarFromCtxFn type which has the signature it sounds like both @jzelinskie and I are hoping for. The labels returned from this function are applied as exemplars to the observed metrics, rather than labels.

go-grpc-middleware/providers/prometheus/server_options.go

Line 12 in bea7c0e

type exemplarFromCtxFn func(ctx context.Context) prometheus.Labels

I suppose I'm proposing adding an additional Option like WithLabelsFromContext rather than WithExemplarFromContext to support labeling a metric dynamically with request context.

@johanbrandhorst If this sounds reasonable to you I'll move forward w/ a PR, but just want to confirm first before spending any time on it. Please let me know if I need to clarify anything.

[johanbrandhorst]

Sounds good, a PR with some tests, ideally some examples for use and some basic docs would be great, thank you!