PMM Client Support for Remote Write to External Mimir with mTLS and Metric Relabeling

[matan77]

Description

Summary

Enhance Percona PMM Client to support pushing metrics via remote_write to external Grafana Mimir endpoints, including mTLS (client certificate authentication) and custom label injection/overrides for built-in exporters such as Node Exporter and PostgreSQL Exporter.

---

Current Problem

At the moment, PMM Client tightly couples metrics ingestion to the PMM server stack. This creates limitations for users who:

• Operate hybrid observability stacks (PMM + Grafana Mimir / Cortex / Thanos)
• Need to send metrics to centralized multi-tenant Prometheus-compatible backends
• Require secure outbound connectivity using mTLS
• Want to enforce consistent labeling standards across all metrics sources

Additionally:

• There is no native, supported mechanism in PMM Client to configure remote_write destinations
• There is no standardized way to attach or override labels at scrape/exporter level inside PMM
• TLS authentication is limited and does not support client certificate-based authentication (mTLS) for remote endpoints

---

Proposed Feature

1. Remote Write Support in PMM Client

Add configuration support in PMM Client for one or more remote_write endpoints:

• Compatible with Prometheus remote_write protocol
• Support multiple endpoints (fan-out)
• Configurable per environment

Example (conceptual):

remote_write:
  - url: https://mimir.example.com/api/v1/push
    tls_config:
      ca_file: /etc/pmm/certs/ca.crt
      cert_file: /etc/pmm/certs/client.crt
      key_file: /etc/pmm/certs/client.key
      insecure_skip_verify: false

---

2. mTLS Support for Remote Write

Enable full TLS client authentication:

• CA validation
• Client certificate authentication
• Private key support
• Optional certificate rotation compatibility

This is critical for:

• Regulated environments (financial, healthcare, enterprise)
• Zero-trust network architectures
• Secure multi-tenant ingestion pipelines

---

3. Custom Labels Injection / Relabeling

Allow adding or overriding labels for metrics produced by built-in exporters:

Use cases:

• Add environment, cluster, region, team, service
• Enforce naming consistency across all PMM deployments
• Enable multi-cluster aggregation in Mimir

Example:

global_labels:
  environment: "prod"
  cluster: "db-cluster-1"

exporter_labels:
  node_exporter:
    role: "infra"
  postgres_exporter:
    db_tier: "primary"

Or Prometheus-style relabeling:

metric_relabel_configs:
  - source_labels: [__name__]
    target_label: environment
    replacement: prod

---

What Problem This Solves

1. Breaks PMM isolation from modern observability stacks

Users can integrate PMM into:

• Grafana Mimir
• Thanos
• Cortex
• Other Prometheus-compatible backends

2. Eliminates need for sidecar exporters or external Prometheus agents

Today users often deploy:

• Additional Prometheus instances just to forward metrics
• Sidecars or custom exporters for relabeling
  This feature removes that overhead.

3. Enables secure enterprise-grade metric forwarding

mTLS support ensures:

• Secure identity-based ingestion
• Compliance with strict security policies
• Avoidance of static bearer tokens

4. Standardizes labeling across infrastructure

Global and exporter-level labels allow:

• Better aggregation in multi-tenant systems
• Easier filtering in Mimir queries
• Cleaner observability taxonomy

---

Benefits to Users

• Seamless integration of PMM with modern Prometheus ecosystems
• Reduced infrastructure complexity (no extra Prometheus relay layer)
• Strong security posture with mTLS support
• Improved observability consistency via enforced labeling
• Easier migration paths from PMM-only setups to hybrid observability architectures

---

Expected Improvement Over Current Behavior

Current state:

• PMM is primarily a closed metrics pipeline
• Limited or no native remote_write extensibility
• No structured way to enforce global labels across exporters
• TLS support is insufficient for mTLS-based systems

Desired state:

• PMM Client behaves as a first-class Prometheus remote_write agent
• Secure, configurable, multi-destination metric forwarding
• Full control over labeling strategy per exporter and globally

---

Additional Notes

• Backward compatibility must be preserved (remote_write disabled by default)
• Configuration should integrate cleanly with existing PMM Client config management
• Metrics duplication prevention and retry buffering behavior should follow Prometheus best practices
  write this as plain text dont change any word

Suggested solution

No response

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct