[FEATURE] Migrate Grafana Agent to Grafana Alloy

[merhard]

Is your feature request related to a problem? Please describe.
Grafana Agent has been deprecated and will reach EOL November 1, 2025. It has been replaced with a different tool: Grafana Alloy.

Describe the solution you would like to see
The usage of Grafana Agent in PromEx should be migrated to Grafana Alloy instead. There is a migration guide here: https://grafana.com/docs/alloy/latest/set-up/migrate/from-static/

How would you expect this feature to work
I expect it to be an almost drop-in replacement for the current Grafana Agent code. The current Grafana Agent config file will need to be migrated to the Grafana Alloy format. There seems to be a CLI tool that can migrate existing configs: https://grafana.com/docs/alloy/latest/reference/cli/convert/#static

Additional context
https://grafana.com/docs/agent/latest/

[btribouillet]

We faced the same GrafanaAgent deprecation and took a different route: instead of migrating to Alloy, we replaced the binary entirely with a pure-Elixir GenServer that pushes metrics via Prometheus remote_write protocol.

How it works:

1. A GenServer calls PromEx.get_metrics/1 on a configurable interval (e.g. 15s)
2. Parses the Prometheus text format into {name, labels, value} tuples
3. Encodes as Protobuf → Snappy compresses → POSTs to the remote_write endpoint (e.g. Grafana Cloud)
4. Includes exponential backoff on failure, label injection (instance, job), and graceful error handling

No external binary, no sidecar — just standard OTP supervision. It also makes the ETSCronFlusher less critical since get_metrics/1 flushes ETS as a side effect on every push cycle.

We'd be happy to generalize this into a PR for the fork at Flowganise/prom_ex as a built-in PromEx.MetricsPusher module — a first-class replacement for PromEx.GrafanaAgent that works with any Prometheus remote_write compatible backend.

[akoutmos]

I am actually working on supporting alloy as of this week (needed it for a customer project). So support for Alloy should be out next week.

As for supporting remote_write directly through Elixir without the need for Alloy/Agent @btribouillet, I would definitely be interested in merging in that functionality. I would want to keep support for Agent/Alloy as many people use the separate binary to also collect logs and traces so it does offer some utility outside of just metrics.

[btribouillet]

Following up on the GenServer approach. I'd like to understand your vision for how this fits alongside Alloy.

I was thinking of offering both strategies under a single metrics_push: config (:remote_write for pure Elixir, :alloy for the binary), but I'm not sure both are needed. The GenServer covers the metrics-only use case.

A couple of questions:

Do you see PromEx managing the Alloy binary like it did with Grafana Agent, or would the simpler in-process approach be enough?

[merhard]

support for Alloy should be out next week.

@akoutmos just checking in on how the Alloy support is coming. I'm now seeing a "Critical" vulnerability report on the latest Agent binary due to an outdated Go gRPC dependency that causes my CI pipeline to fail: GHSA-p77j-4mvh-x3m3

[gf3]

Throwing in my 2 cents here—we run our staging and production environments in a Kubernetes cluster where we already run Alloy et al to ship logs, metrics, traces, etc... to our observability provider. Therefore it's less useful for us (and likely others with this type of deployment) to have PromEx run its own Alloy and instead would be easier to connect to an existing installation.