prometheus: scrape per-track metrics per namespace

/metrics/track takes one namespace per request and rejects a match wider
than its limit, so an unscoped scrape fails once the relay's total track
count passes the limit and takes every namespace with it. Scoping per
namespace keeps working until a single namespace passes it.

Namespaces change as events start and end, so ns-targets walks the relay's
namespace tree and writes the file_sd target list; Prometheus rereads it
without a restart. Encoding matches the endpoint's safe form.

Author: gmarzot

docker/docker-compose.yml

@@ -202,6 +202,7 @@ services:
     mem_limit: ${PROMETHEUS_MEM:-2g}
     volumes:
       - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - prometheus-targets:/etc/prometheus/targets:ro
       # TSDB backing store. Defaults to the named volume (persists across
       # redeploys — `compose down` without -v keeps it). Set PROMETHEUS_DATA_DIR
       # to an absolute host path to pin it on a specific/larger disk instead.
@@ -315,6 +316,29 @@ services:
         max-size: "10m"
         max-file: "3"
 
+  # Writes one Prometheus file_sd target per live namespace, read from the
+  # relay's namespace tree. /metrics/track takes a single namespace per
+  # request, and namespaces come and go with events, so the list cannot be
+  # static.
+  ns-targets:
+    container_name: moqx-ns-targets
+    image: python:3.12-alpine
+    restart: unless-stopped
+    profiles: [stats]
+    command: ["python3", "/app/namespace-targets.py"]
+    environment:
+      MOQX_STATE_URL: http://moqx:${MOQX_ADMIN_PORT:-8000}/state
+      MOQX_TARGETS_PATH: /targets/namespaces.json
+      MOQX_TARGETS_INTERVAL: "30"
+    volumes:
+      - ./prometheus/namespace-targets.py:/app/namespace-targets.py:ro
+      - prometheus-targets:/targets
+    logging:
+      driver: json-file
+      options:
+        max-size: "10m"
+        max-file: "3"
+
   # Turns the relay admin /state JSON into per-track Prometheus series (the
   # aggregate /metrics has no per-track labels). Prometheus scrapes it with a
   # target= param pointing at the relay /state (see the moqx-state job).
@@ -334,6 +358,7 @@ services:
         max-file: "3"
 
 volumes:
+  prometheus-targets:
   moqx-coredumps:
   prometheus-data:
   grafana-data:

docker/prometheus/README.md

@@ -2,31 +2,40 @@
 
 ## Per-track metrics (`moqx-track`)
 
-`/metrics/track` takes optional `service`, `namespace`, `track` and `limit`
-parameters; with none of them it reports every live track, so a single scrape
-covers the relay.
+`/metrics/track` reports counters per live track. It takes one namespace per
+request, so each namespace is scraped separately.
 
-`limit` is a guard rather than a selector: more live tracks than the limit
-returns 400, so an over-wide scrape fails visibly instead of graphing a series
-set that reshuffles between scrapes. Ceiling is
-`admin.track_metrics_endpoint_max_limit` (1000).
+Scoping per namespace is not just tidiness. `limit` is a guard rather than a
+selector: a request matching more tracks than the limit returns 400 instead of
+truncating, because an arbitrary subset would give Prometheus a series set that
+reshuffles between scrapes. An unscoped scrape therefore fails as soon as the
+relay's *total* track count passes the limit, and takes every namespace with
+it, while per-namespace scrapes keep working until a *single* namespace passes
+it. Ceiling is `admin.track_metrics_endpoint_max_limit` (1000).
 
-Past that ceiling the scrape splits by namespace prefix, one job per prefix,
-with `/state.namespace_tree` generating the target list into a `file_sd` file:
+Counting only happens when `admin.track_metrics_enabled` is true (the default);
+with it false the endpoint returns 503 rather than an empty scrape that would
+read as "no live tracks".
 
-    file_sd_configs:
-      - files: ['/etc/prometheus/targets/namespaces.json']
-        refresh_interval: 30s
+## Target generation (`ns-targets`)
 
-Label values use the moq-transport encoded form — tuple elements joined by
-`-`, other bytes as `.<hex>`:
+Namespaces come and go with events, so the target list cannot be static.
+`namespace-targets.py` walks the relay's `/state` namespace tree every 30s and
+writes one target per namespace to a file Prometheus rereads without a restart.
+It collects every node carrying a namespace rather than only the leaves, since
+tracks can be published at any depth.
+
+Targets are namespace values in the moq-transport safe form — `[A-Za-z0-9_]`
+passes through, every other byte becomes `.<hex>`, tuple elements join with
+`-`:
 
     moq-test/interop           ->  moq.2dtest-interop
     conf.example.com / room 1  ->  conf.2eexample.2ecom-room.201
 
-Series exist only for live tracks: they disappear when a track ends and restart
-from zero if it returns, so counters need `increase()`/`rate()` rather than
-raw deltas across a track's lifetime.
+Relabelling turns each target into the `namespace` query parameter and points
+the scrape at the relay. The unencoded namespace is kept as `moqx_namespace`
+for display, since the encoded form is what lands in the metric labels.
 
-Counting is installed only when `admin.track_metrics_enabled` is true (default);
-with it false the endpoint returns 503.
+Series exist only while a track is live: they disappear when it ends and
+restart from zero if it returns, so counters need `rate()`/`increase()` rather
+than differences taken across a track's lifetime.

docker/prometheus/__pycache__/namespace-targets.cpython-310.pyc

@@ -0,0 +1,92 @@
+#!/usr/bin/env python3
+"""Generate Prometheus file_sd targets for per-track metric scrapes.
+
+/metrics/track takes one namespace per request and rejects a match wider than
+its limit, so each namespace is scraped separately. This walks the relay's
+namespace tree and writes one target per namespace; Prometheus rereads the file
+without a restart.
+
+Namespaces are written in the moq-transport safe form the endpoint expects:
+[A-Za-z0-9_] passes through, every other byte becomes .<hex>, and tuple
+elements are joined with '-'.
+"""
+import json
+import os
+import sys
+import tempfile
+import time
+import urllib.request
+
+STATE_URL = os.environ.get("MOQX_STATE_URL", "http://moqx:8000/state")
+OUT_PATH = os.environ.get("MOQX_TARGETS_PATH", "/targets/namespaces.json")
+INTERVAL = float(os.environ.get("MOQX_TARGETS_INTERVAL", "30"))
+
+_PASS = set(
+    "abcdefghijklmnopqrstuvwxyz" "ABCDEFGHIJKLMNOPQRSTUVWXYZ" "0123456789" "_"
+)
+
+
+def safe_element(element):
+    out = []
+    for byte in element.encode():
+        ch = chr(byte)
+        out.append(ch if ch in _PASS else ".%02x" % byte)
+    return "".join(out)
+
+
+def safe_namespace(tuple_elements):
+    return "-".join(safe_element(e) for e in tuple_elements)
+
+
+def walk(node, found):
+    """Collect every node carrying a namespace, not just the leaves: tracks can
+    be published at any depth."""
+    full = node.get("full_namespace") or []
+    if full:
+        found.append(full)
+    for child in (node.get("children") or {}).values():
+        walk(child, found)
+
+
+def namespaces():
+    with urllib.request.urlopen(STATE_URL, timeout=10) as resp:
+        state = json.load(resp)
+    found = []
+    for service in (state.get("services") or {}).values():
+        tree = service.get("namespace_tree")
+        if tree:
+            walk(tree, found)
+    # A namespace can appear under more than one service.
+    return sorted({tuple(ns) for ns in found})
+
+
+def write(path, entries):
+    payload = [
+        {"targets": [safe_namespace(ns)], "labels": {"moqx_namespace": "/".join(ns)}}
+        for ns in entries
+    ]
+    body = json.dumps(payload, indent=2) + "\n"
+    if os.path.exists(path) and open(path).read() == body:
+        return False
+    # Rename into place so Prometheus never reads a partial file.
+    directory = os.path.dirname(path) or "."
+    fd, tmp = tempfile.mkstemp(dir=directory)
+    with os.fdopen(fd, "w") as handle:
+        handle.write(body)
+    os.replace(tmp, path)
+    return True
+
+
+def main():
+    while True:
+        try:
+            entries = namespaces()
+            if write(OUT_PATH, entries):
+                print("wrote %d namespace target(s)" % len(entries), flush=True)
+        except Exception as exc:  # keep polling: the relay restarts
+            print("namespace target refresh failed: %s" % exc, file=sys.stderr, flush=True)
+        time.sleep(INTERVAL)
+
+
+if __name__ == "__main__":
+    main()

docker/prometheus/namespace-targets.py

@@ -73,16 +73,24 @@ scrape_configs:
       - target_label: __address__
         replacement: json-exporter:7979
 
-  # Per-track counters from the relay (openmoq/moqx#533). Every parameter is
-  # optional, so one scrape covers all namespaces. limit is a guard, not a
-  # top-N: more live tracks than the limit returns 400 rather than a series
-  # set that reshuffles between scrapes.
+  # Per-track counters from the relay (openmoq/moqx#533). One namespace per
+  # request: an unscoped scrape covers everything only until the total exceeds
+  # the limit, at which point it 400s and takes every namespace down with it,
+  # while per-namespace scrapes keep working until a single namespace exceeds
+  # it. Targets are namespaces in the safe form, generated from the relay's
+  # namespace tree by ns-targets and relabelled into the query.
   - job_name: moqx-track
     metrics_path: /metrics/track
     scrape_interval: 15s
     params:
       limit: ['1000']
-    static_configs:
-      - targets: ['moqx:8000']
-        labels:
-          instance: moqx-relay
+    file_sd_configs:
+      - files: ['/etc/prometheus/targets/namespaces.json']
+        refresh_interval: 30s
+    relabel_configs:
+      - source_labels: [__address__]
+        target_label: __param_namespace
+      - target_label: __address__
+        replacement: moqx:8000
+      - target_label: instance
+        replacement: moqx-relay