Forwarding Rules from Local Mimir to Remote Mimir (different cluster)

[rafaelpissolatto]

Hi There!

I'm working on a use case where I need to forward PrometheusRules from a Local Mimir setup (running on in the same cluster) to a Remote Mimir setup (different cluster/remote), where the rules should evaluate using metrics from the Local Mimir instance (see the example of PrometheusRules where . I've validated that tenant federation doesn't work across different instances, and I'd like see if I have any other options.

The metrics or some rules will be stored via Alloy to a local Mimir in the same cluster, but for some cases, we want to forward the same rule ingested by the local Mimir to another remote Mimir which will have diff retention and other internal requirements

(The goal is to forward rules with prefixes like my_prefix_3m:my_metric from Local Mimir to Remote Mimir, where are being ingested into multiple tenants in the Local Mimir (e.g., tenant-sys, tenant-app, tenant-edge)

Maybe pretty similar to what GEM offers: https://grafana.com/docs/enterprise-metrics/latest/manage/cluster-query-federation/

PrometheusRules like (Examp:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: test-rules
  namespace: <namespace>
  annotations:
spec:
  groups:
    - name: test.rules
      interval: 15s
      rules:
       # record rule pointing to a raw query
        - record: my_prefix_3m:loki_request_duration_1m
          expr: sum by (pod) (rate(loki_request_duration_seconds_sum[1m]))
          labels:
            test: "my_labels"
       # record rule pointing to another rule 
        - record: my_prefix_3m:pod_department_memory_requests_namespace_running
          expr: pod_department_memory_requests_by_namespace{pod_state="running"}
          labels:
            test: "my_labels"

I was trying out using Grafana Alloy with:

• mimir.rules.kubernetes to sync rules to Local Mimir per tenant (+ annotation: monitoring.grafana.com/source_tenants: <context>, but it seems working only for the same Mimir setup with multiple tenants)
• prometheus.scrape Attempting to scrape evaluated metrics from Local Mimir and forward to Remote Mimir as a "metric only" (blocked by Mimir API limitations, not able to read as prometheus metrics. Found /v1/read but prometheus.scrape expects a non-json responses) - metric should be enough for us since we want to use the Remote Mimir to centralize visualization at first moment

Thank you