Add alternative prometheus remote write implementation to `prometheuesremotewriteexporter`

[mattdurham]

Component(s)

exporter/prometheusremotewrite

Is your feature request related to a problem? Please describe.

For Grafana Alloy I have been working on an alternative approach to the prometheus remote write WAL.

Describe the solution you'd like

Would it be reasonable to add the this new remote implementation into prometheusremotewriteexporter, the config options would be significantly different. Than the current WAL config so would need to only declare one or the other. Its currently experimental in Alloy but we have been using it internally and alongside several other users.

We haven't had any issues scaling it to a few million series writes a second, also looking into support for the new RW format that would mean only updating the library along with general support.

The underlying code can be found at https://github.com/grafana/walqueue

Describe alternatives you've considered

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/prometheusremotewrite: @Aneurysm9 @rapphil @dashpole @ArthurSens

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

A few questions/points:

• What are the benefits of the new WAL compared to the existing WAL implementation? We should spell those out clearly.
• We should not maintain multiple WAL implementations long-term. So if we plan to adopt a new WAL, we should plan for it to replace the existing WAL implementation.
• The above means we need a migration plan. If someone updates their collector version without changing configuration, we should not drop telemetry stored in the "legacy" WAL.
• I the WAL was initially implemented because we needed in-order points and the exporter helper's persistent queue doesn't preserve order.
  • Given Prometheus has started supporting OOO samples, do we still need a WAL? (I think the answer is yes, but we should think this through.
  • Should the WAL be added to exporterhelper?

The effort to add support for PRW 2.0 is being worked on by @jmichalek132. I would sync up with him on that. If we can keep that effort decoupled from the WAL implementation, that would be preferable.