metrics: add dyncfg-driven export filter for environmentd metrics

[bosconi]

Motivation

In INC-1252 a single environmentd /metrics response (~58 MB, ~450k samples) exceeded the AMP scraper's 50 MiB body-size limit, so every scrape failed and all environmentd metrics for the environment disappeared for hours, twice. Many families carry one series per catalog object, persist shard, replica or cluster, so the response grows with the catalog, and until now the only lever was a code change and a rollout.

Tracking issue: DB-199. Design: doc/developer/design/20260902_metrics_export_filter.md in this PR.

Description

Adds a gather-time postprocessor on the metrics registry, installed once per environmentd instance from mz_environmentd::serve, that drops families and scopes series at export time according to four new dyncfgs. All default to no filtering.

dyncfg	Effect
metrics_export_disabled_families	Comma-separated family names, trailing * matches a prefix. Matching families are removed.
metrics_export_cluster_allowlist	When set, series carrying instance_id, cluster_id or compute_instance are exported only for the listed clusters.
metrics_export_replica_allowlist	Same for replica_id.
metrics_export_max_series_per_family	A family with more exported samples than this is dropped whole rather than truncated, so a dashboard sees an absent family instead of a partial one that looks complete. Zero disables.

The filter records what it did in mz_metrics_export_series{family}, mz_metrics_export_dropped_series_total{family,reason} and mz_metrics_export_encoded_bytes. Its own families are exempt from filtering so the operator can always see what was dropped. The per-family sample gauge and the encoded-bytes gauge are the leading indicators we lacked during the incident.

Design notes:

• Counts are exported samples, as the text encoder emits them, not label sets. A histogram series expands to one line per bucket plus +Inf, _count and _sum, and histogram-heavy families dominated the incident.
• Configuration is read from the live system dyncfg set that environmentd already holds and the storage controller already updates on every system-config change, so there is no catalog round-trip per scrape, no process-global state, and no separate update hook. Raw values are re-parsed only when they change.
• Filtering is export-side only. Filtered series still exist in the registry, so re-enabling a family restores its current values, and the process still pays for maintaining them. Emission-side gating per family (persist shard metrics, per-collection wallclock lag, the application_name label) is follow-up work in DB-199.
• clusterd does not install the filter, so clusterd exports are unaffected. /metrics/public gathers environmentd's own families through the filter and then merges clusterd-sourced series it fetches separately, so the filter governs environmentd's own series only.

Tests

• Unit tests in mz_metrics::export_filter for list parsing, each rule, the cap, histogram sample counting, self-metric exemption, stale gauge children, and live dyncfg updates.
• test_metrics_export_filter in src/environmentd/tests/server.rs flips each dyncfg via ALTER SYSTEM on a running environmentd and scrapes /metrics.
• CI system-parameter defaults run the filter path with a non-existent probe family and a high cap, so the postprocessor executes in every mzcompose-based test without changing what tests observe. The drop branches are covered by the unit and integration tests, not by the CI defaults.

Tips for reviewer

Start with src/metrics/src/export_filter.rs; everything else is wiring. The design doc covers the alternatives considered, including the process-global variant this PR replaced.

Checklist

• This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
• This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
• If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
• If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label.
• If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.

🤖 Generated with Claude Code