Metrics whitelist for alloy (#418)

## Summary

Comprehensive metrics cost control for the monitoring module — reduces
cardinality, halves DPM billing, fixes broken scraping, and adds
bad-node detection capability.

### Scrape interval standardization (halves DPM)
- **All monitors now use 60s scrape interval** (Grafana recommendation)
— directly halves data points per minute (DPM), reducing Grafana Cloud
billing (billed at 95th percentile)
- Changed: kubelet, cadvisor, KSM, DCGM, ARC controller, Karpenter,
BuildKit, BuildKit HAProxy, Harbor, git-cache-central, git-cache
DaemonSet, node-compactor
- Exceptions: arc-listeners (3m), node-exporter/apiserver/coredns
(already 60s)

### Metrics cardinality reduction
- **Disable kubelet `/metrics/probes`** (`probes: false`) — drops ~1800
`prober_probe_*` series with no alerting value
- **Karpenter**: replace broad regex with precise keep list (7 metrics)
— saves ~400-500 series by dropping all histogram buckets and low-value
gauges
- **API server**: switch from drop-list to keep-whitelist
(`apiserver_request_total`, `apiserver_request_terminations_total`)
- **ARC controller**: add keep-whitelist (`gha_controller_.*`,
`controller_runtime_reconcile_errors_total`)
- **BuildKit**: drop `go_.*`, `process_.*`, `promhttp_.*`, and all
`_bucket` histograms
- **BuildKit HAProxy**: switch to keep-list (4 metrics:
`haproxy_server_status`, `haproxy_server_current_sessions`,
`haproxy_server_connection_errors_total`,
`haproxy_backend_current_sessions`)
- **Harbor**: drop `go_.*`, `process_.*`, `promhttp_.*`
- **KSM**: explicit `keep` allowlist (RE2-compatible, no lookahead):
  - daemonset: 4 health status metrics only
- deployment: 5 core metrics (replica status + conditions + desired
count)
  - namespace/node/statefulset/pv/hpa/job: all (low cardinality)
  - pod: `kube_pod_info` + error indicators + non-routine status reasons
  - Drop successful terminations (exit code 0, reason Completed)
  - Drop routine status reasons (Shutdown, NodeAffinity)
- **Node-exporter**: keep-whitelist for CPU (idle/iowait only) and
memory metrics
- **Kubelet**: keep-whitelist for `kubelet_running_pods/containers` and
`kubelet_node_name`
- **Alloy cost_control**: drop
`go_.*|process_.*|promhttp_.*|prometheus_operator_.*` globally, drop
`kubernetes_feature_enabled`, drop ARC histogram buckets

### Fix KSM regex (RE2 compatibility)
- Replace `(?!...)` negative lookahead with explicit `keep` allowlist —
RE2 (used by Prometheus/Alloy relabeling) does not support lookahead
syntax, causing all `kube_*` metrics to be silently dropped

### Bad node detection
- **Keep `kube_pod_info`** in KSM metrics — provides `node` label to
correlate pod failures with specific nodes:
  ```promql
  count(

kube_pod_container_status_last_terminated_exitcode{container_exit_code!="0"}
    * on(namespace, pod) group_left(node) kube_pod_info
  ) by (node)
  ```

### Fix arc-listeners PodMonitor (was returning no data)
- **Namespace**: `arc-runners` → `arc-systems` (listener pods run in
arc-systems)
- **Label selector**: → `app.kubernetes.io/component:
runner-scale-set-listener`
- **Add metricRelabelings**: keep only important listener metrics, drop
`instance` label
- **Scrape interval**: 30s → 3m (reduce load, listener metrics change
slowly)

### Documentation
- Add comprehensive "What we collect" section to CLAUDE.md per source
- Document 60s scrape interval standard, KSM keep-list, HAProxy
keep-list, status_reason filtering

Author: yangw-dev

osdc/modules/monitoring/CLAUDE.md

@@ -0,0 +1,24 @@
+# modules/monitoring/
+
+## Key rules
+
+- **All scrape intervals must be 60s** (Grafana recommendation — halves DPM billing). Only exception: arc-listeners at 3m. Do not add monitors below 60s without justification.
+- **KSM metricRelabelings use RE2** — no `(?!...)` lookahead support. Use explicit `keep` allowlists instead.
+- **Alloy is the only metrics pipeline** — no local Prometheus. PrometheusRule CRDs are synced to Grafana Cloud via `mimir.rules.kubernetes`.
+
+## Metrics filtering layers
+
+1. **`--metric-allowlist`** (KSM server-side) — controls which resource groups KSM generates
+2. **`metricRelabelings`** (ServiceMonitor/PodMonitor) — `keep` whitelists per source
+3. **Alloy `cost_control`** (before remote_write) — safety net for anything not filtered at source
+
+## Bad node detection
+
+`kube_pod_info` provides pod-to-node mapping. Join with error metrics to find bad nodes:
+
+```promql
+count(
+  kube_pod_container_status_last_terminated_exitcode{container_exit_code!="0"}
+  * on(namespace, pod) group_left(node) kube_pod_info
+) by (node)
+```

osdc/modules/monitoring/deploy.sh

@@ -111,6 +111,8 @@ EOF
     grafana/alloy
 
   rm -f "$ALLOY_OVERRIDE"
+  kubectl rollout restart deployment/alloy -n "$NAMESPACE"
+  kubectl rollout status deployment/alloy -n "$NAMESPACE" --timeout=3m
   echo "Alloy installed — pushing metrics to Grafana Cloud."
 else
   echo "No grafana-cloud-credentials secret found, skipping Alloy (no remote metrics push)."

osdc/modules/monitoring/helm/alloy-values.yaml

@@ -47,35 +47,37 @@ alloy:
       }
 
       // Drop high-cardinality and low-value metrics before remote_write.
+      // Most filtering is done at ServiceMonitor level (keep whitelists).
+      // These rules catch anything that slips through from sources without per-monitor filtering.
       prometheus.relabel "cost_control" {
         forward_to = [prometheus.remote_write.grafana_cloud.receiver]
 
-        // Drop noisy cadvisor metrics (network, tasks, load avg, blkio, spec)
+        // --- KSM: drop low-value metrics ---
         rule {
           action        = "drop"
           source_labels = ["__name__"]
-          regex         = "container_network_(tcp|udp)_usage_total|container_tasks_state|container_cpu_load_average_10s|container_memory_failures_total"
+          regex         = "kube_.*_created|kube_.*_metadata_resource_version|kube_secret_.*|kube_configmap_.*|kube_endpoint_.*|kube_lease_.*"
         }
 
-        // Drop cadvisor lifecycle and spec metrics
+        // --- drop misc high-cardinality metrics ---
         rule {
           action        = "drop"
           source_labels = ["__name__"]
-          regex         = "container_blkio_device_usage_total|container_last_seen|container_start_time_seconds|container_spec_.*"
+          regex         = "kubernetes_feature_enabled"
         }
 
-        // Drop low-value KSM metrics
+        // --- ARC histogram buckets (keep sum/count) ---
         rule {
           action        = "drop"
           source_labels = ["__name__"]
-          regex         = "kube_.*_created|kube_.*_metadata_resource_version|kube_secret_.*|kube_configmap_.*|kube_endpoint_.*|kube_lease_.*"
+          regex         = "gha_job_(execution|startup)_duration_seconds_bucket"
         }
 
-        // Drop ARC histogram buckets (keep sum/count)
+        // --- go/process/promhttp/prometheus_operator internals from any source ---
         rule {
           action        = "drop"
           source_labels = ["__name__"]
-          regex         = "gha_job_(execution|startup)_duration_seconds_bucket"
+          regex         = "go_.*|process_.*|promhttp_.*|prometheus_operator_.*"
         }
       }
 

osdc/modules/monitoring/helm/values.yaml

@@ -30,6 +30,28 @@ kubeScheduler:
 kubeEtcd:
   enabled: false
 
+# --- kubelet + cadvisor ServiceMonitor ---
+kubelet:
+  enabled: true
+  serviceMonitor:
+    interval: 60s
+    # Disable /metrics/probes scraping — generates ~1440 high-cardinality
+    # prober_probe_duration_seconds_bucket series with no alerting value
+    probes: false
+    # cadvisor: keep only pod-level memory metrics (per pod/container series)
+    # - container_memory_working_set_bytes: active memory used by pod; K8s uses this for OOM kill decisions
+    # - container_memory_rss: physical memory (no cache); useful for spotting memory leaks
+    # NOTE: these generate one series per container, so high pod churn (e.g. runners) increases cardinality
+    cAdvisorMetricRelabelings:
+      - action: keep
+        sourceLabels: [__name__]
+        regex: "container_memory_working_set_bytes|container_memory_rss"
+    # kubelet: keep only running pod/container counts
+    metricRelabelings:
+      - action: keep
+        sourceLabels: [__name__]
+        regex: "kubelet_running_(pods|containers)|kubelet_node_name"
+
 # --- Grafana (DISABLED — use Grafana Cloud) ---
 grafana:
   enabled: false
@@ -66,11 +88,48 @@ prometheus-node-exporter:
   prometheus:
     monitor:
       interval: 60s
+      metricRelabelings:
+        # Keep only CPU (idle/iowait) and memory — drop everything else
+        - action: keep
+          sourceLabels: [__name__]
+          regex: "node_cpu_seconds_total|node_memory_MemAvailable_bytes|node_memory_MemTotal_bytes"
+        - action: drop
+          sourceLabels: [__name__, mode]
+          regex: "node_cpu_seconds_total;(user|system|nice|irq|softirq|steal)"
 
 # --- kube-state-metrics ---
 kube-state-metrics:
   extraArgs:
-    - --metric-allowlist=kube_(daemonset|deployment|pod|namespace|node|statefulset|persistentvolume|horizontalpodautoscaler|replicaset|job)_.+
+    - --metric-allowlist=kube_(daemonset|deployment|pod|namespace|node|statefulset|persistentvolume|horizontalpodautoscaler|job)_.+
+  prometheus:
+    monitor:
+      interval: 60s
+      metricRelabelings:
+        # Keep only the KSM metrics we actually use (single rule — multiple keeps are ANDed).
+        # RE2 engine (no lookahead), so we use an explicit allowlist.
+        #   daemonset: health status (desired/ready/available/unavailable)
+        #   deployment: replica status + conditions + desired count
+        #   namespace/node/statefulset/pv/hpa/job: all metrics (low cardinality)
+        #   pod: info (node mapping), error indicators, status reasons
+        - action: keep
+          sourceLabels: [__name__]
+          regex: "kube_daemonset_status_(desired_number_scheduled|number_ready|number_available|number_unavailable)|kube_deployment_status_(replicas_ready|replicas_available|replicas_unavailable|condition)|kube_deployment_spec_replicas|kube_(namespace|node|statefulset|persistentvolume|horizontalpodautoscaler|job)_.*|kube_pod_info|kube_pod_container_status_last_terminated_reason|kube_pod_container_status_last_terminated_exitcode|kube_pod_status_reason"
+        # Drop low-value node metrics
+        - action: drop
+          sourceLabels: [__name__]
+          regex: "kube_node_status_addresses"
+        # Drop successful terminations (only keep actual errors)
+        - action: drop
+          sourceLabels: [__name__, reason]
+          regex: "kube_pod_container_status_last_terminated_reason;Completed"
+        - action: drop
+          sourceLabels: [__name__, container_exit_code]
+          regex: "kube_pod_container_status_last_terminated_exitcode;0"
+        # Drop routine pod status reasons (spot reclaim, scheduling changes)
+        # Keep only non-routine errors: Evicted, NodeLost, UnexpectedAdmissionError
+        - action: drop
+          sourceLabels: [__name__, reason]
+          regex: "kube_pod_status_reason;(Shutdown|NodeAffinity)"
   tolerations:
     - key: CriticalAddonsOnly
       operator: Equal

osdc/modules/monitoring/kubernetes/monitors/dcgm-servicemonitor.yaml

@@ -14,7 +14,7 @@ spec:
       app.kubernetes.io/name: dcgm-exporter
   endpoints:
     - port: metrics
-      interval: 30s
+      interval: 60s
       metricRelabelings:
         - action: labeldrop
           regex: UUID

osdc/modules/monitoring/kubernetes/monitors/podmonitors/arc-listeners.yaml

@@ -8,11 +8,20 @@ metadata:
 spec:
   namespaceSelector:
     matchNames:
-      - arc-runners
+      - arc-systems
   selector:
     matchLabels:
-      actions.github.com/scale-set-namespace: arc-runners
+      app.kubernetes.io/component: runner-scale-set-listener
   podMetricsEndpoints:
     - port: metrics
-      interval: 30s
+      interval: 3m
       scrapeTimeout: 10s
+      metricRelabelings:
+        # Keep only operationally important listener metrics
+        # Drops low-value gauges (idle/busy/registered/min/max runners)
+        - action: keep
+          sourceLabels: [__name__]
+          regex: "gha_assigned_jobs|gha_completed_jobs_total|gha_started_jobs_total|gha_running_jobs|gha_job_execution_duration_seconds_(sum|count)|gha_job_startup_duration_seconds_(sum|count)"
+        # Drop instance label — pod IP changes on restart, adds no value
+        - action: labeldrop
+          regex: "instance"

osdc/modules/monitoring/kubernetes/monitors/podmonitors/git-cache-daemonset.yaml

@@ -14,4 +14,4 @@ spec:
       app: git-cache-warmer
   podMetricsEndpoints:
     - port: metrics
-      interval: 30s
+      interval: 60s

osdc/modules/monitoring/kubernetes/monitors/servicemonitors/apiserver.yaml

@@ -24,20 +24,7 @@ spec:
         serverName: kubernetes
         insecureSkipVerify: false
       metricRelabelings:
-        # Drop Go runtime metrics (high cardinality, low operational value)
-        - action: drop
+        # Keep only essential API server health metrics, drop everything else
+        - action: keep
           sourceLabels: [__name__]
-          regex: "go_.*"
-        # Drop process metrics (CPU/memory of the API server process itself)
-        - action: drop
-          sourceLabels: [__name__]
-          regex: "process_.*"
-        # Drop internal workqueue metrics (very noisy, many queues)
-        - action: drop
-          sourceLabels: [__name__]
-          regex: "workqueue_.*"
-        # Drop high-cardinality histogram buckets (keep _sum/_count for averages)
-        - action: drop
-          sourceLabels: [__name__]
-          regex: >-
-            apiserver_request_duration_seconds_bucket|apiserver_request_sli_duration_seconds_bucket|apiserver_response_sizes_bucket|apiserver_watch_events_sizes_bucket|apiserver_admission_controller_admission_duration_seconds_bucket
+          regex: "apiserver_request_total|apiserver_request_terminations_total"

osdc/modules/monitoring/kubernetes/monitors/servicemonitors/arc-controller.yaml

@@ -14,4 +14,9 @@ spec:
       app.kubernetes.io/name: gha-rs-controller
   endpoints:
     - port: metrics
-      interval: 30s
+      interval: 60s
+      metricRelabelings:
+        # Keep only ARC operational metrics and reconcile errors — drop everything else
+        - action: keep
+          sourceLabels: [__name__]
+          regex: "gha_controller_.*|controller_runtime_reconcile_errors_total"

osdc/modules/monitoring/kubernetes/monitors/servicemonitors/buildkit-haproxy.yaml

@@ -15,4 +15,9 @@ spec:
       app.kubernetes.io/component: load-balancer
   endpoints:
     - port: metrics
-      interval: 30s
+      interval: 60s
+      metricRelabelings:
+        # Keep only operationally important HAProxy metrics
+        - action: keep
+          sourceLabels: [__name__]
+          regex: "haproxy_server_status|haproxy_server_current_sessions|haproxy_server_connection_errors_total|haproxy_backend_current_sessions"