Replacing `github.com/rcrowley/go-metrics` with `github.com/prometheus/client_golang`?

[andygrunwald]

While working on #438, I came across https://github.com/rcrowley/go-metrics.

I came across two areas:

Shortcomings

1) Maintenance

See rcrowley/go-metrics#291
The project github.com/rcrowley/go-metrics is no longer maintained; the last commit was 5 years ago.

2) Support for proper labels / tags

https://github.com/rcrowley/go-metrics as well as https://github.com/deathowl/go-metrics-prometheus doesn't seem to support labels.
See rcrowley/go-metrics#135 and deathowl/go-metrics-prometheus#6

Currently, this is represented in the Metrics client middleware as ... (using Prometheus as output format):

# HELP sourcectl_worker_github_rate_used_installation:12345_ github.rate.used[installation:12345]
# TYPE sourcectl_worker_github_rate_used_installation:12345_ gauge
sourcectl_worker_github_rate_used_installation:12345_ 1

In a prometheus native way, this would look like

# HELP sourcectl_worker_github_rate_used_installation:12345_ github.rate.used[installation:12345]
# TYPE sourcectl_worker_github_rate_used_installation:12345_ gauge
sourcectl_worker_github_rate_used{installation=12345} 1

If you want to expose the installation as label, this is a different question, as this is (depending on your use case) a high cardinality value (e.g. if you run it through a lot of installation).

Another shortcoming is the missing / or more unhelpful HELP text.

rcrowley/go-metrics is more than just Prometheus

I know that rcrowley/go-metrics is a generic metric implementation and not Prometheus-specific.
In my setup, I am using Prometheus and the not existing label support is a drawback here.
I do understand that other users of this lib (or even Palantir itself) may not have a Prometheus setup.

Own Middleware or PrometheusClientMetrics Middleware

It is possible for me to write my own Middleware and also contribute it back to this project.
Maybe as a separate PrometheusClientMetrics Middleware.
This would not break the existing go-metrics usage.
It could also be that you decide to release a new major version and run with the breaking change. Here, I do not have insights how you think about it (and what effect this would have on your internal implementations).

I would like to get a bit of feedback on this from you and your thoughts around this.

[bluekeyes]

Thanks for raising an issue for this, as it's something we've talked about internally as well.

To fix the immediate problem with metric labels, take a look at https://pkg.go.dev/github.com/palantir/go-baseapp@v0.5.2/appmetrics/emitter/prometheus. This is a prometheus.Collector implementation for rcrowley/go-metrics that supports the convention of including labels in square brackets as part of the metric name. It should be usable independently from the other stuff in go-baseapp.

Compared to other existing Prometheus integrations for go-metrics, it also implements prometheus.Collector instead of polling the registry at a fixed schedule and correctly exports metrics.Historgram metrics as Prometheus summaries, rather than histograms. It doesn't solve the problem with help text though.

---

As a bit of history, when we first wrote this library, we were using statsd (Datadog) for metrics and building alongside teams that were used to using Dropwizard metrics in Java. Adopting rcrowley/go-metrics provided a similar interface and let us export in whatever format we wanted. That proved useful when we migrated to Grafana/Prometheus two years ago, because we could add new exporters without changing how all of our applications recorded metrics.

Fully removing rcrowley/go-metrics would be a big project for us, as we have many internal applications that would need to be updated to either fully use the new metrics interface or to unify the metrics interface in go-githubapp with the existing go-metrics usage in the rest of the application. Realistically, this isn't something we'd prioritize unless it solved a problem or introduced compelling new functionality.

I'm not too concerned with the lack of development in go-metrics (it seems to be complete and works as advertised), but recognize that it has limitations and that Prometheus and other tools are the de-facto standards now. My hope was that the ability to add your own middleware would allow custom metrics implementations, but I'm open to discussing ways that go-githubapp could make this easier.

I'm hesitant to add a Prometheus-specific metrics function in the library, as it sets a precedent for including other backend-specific functions. In general, I'd like to avoid the library making any specific recommendations here. For instance, if I were writing this again today, I'd probably use the log/slog package and let people provide their own handler rather than use zerolog. Unfortunately, there's no equivalent metrics support in the standard library.

OpenTelemetry might be an option as an alternative interface layer that is more closely aligned with the Prometheus way of doing things. We use this for tracing in some other places and it could make sense to adopt it for metrics as well. That could probably live alongside the existing go-metrics implementation, at least to start.