prometheus: single scrape for per-track metrics

Rework of #533 made every query parameter optional, so one
target covers all namespaces instead of a per-namespace list. Namespace
splitting is only needed past the endpoint's limit ceiling; noted in the
README along with the label encoding and the live-tracks-only lifetime.

Author: gmarzot

docker/prometheus/README.md

@@ -2,30 +2,31 @@
 
 ## Per-track metrics (`moqx-track`)
 
-`/metrics/track` requires a `namespace` parameter, so every namespace to be
-graphed is a scrape target that relabelling turns into the query parameter.
-Namespace values use the moq-transport encoded form — tuple elements joined by
-`-`, other bytes as `.<hex>`:
-
-    moq-test/interop   ->  moq.2dtest-interop
-    conf.example.com / room 1  ->  conf.2eexample.2ecom-room.201
+`/metrics/track` takes optional `service`, `namespace`, `track` and `limit`
+parameters; with none of them it reports every live track, so a single scrape
+covers the relay.
 
-`limit` is a guard rather than a selector: a namespace holding more tracks than
-the limit returns 400, so an over-wide scrape fails visibly instead of graphing
-a series set that reshuffles between scrapes. Ceiling is
-`admin.track_metrics_max_limit` (1000).
+`limit` is a guard rather than a selector: more live tracks than the limit
+returns 400, so an over-wide scrape fails visibly instead of graphing a series
+set that reshuffles between scrapes. Ceiling is
+`admin.track_metrics_endpoint_max_limit` (1000).
 
-### Generating the target list
-
-The static list needs maintaining by hand. `/state.namespace_tree` already
-enumerates live namespaces, so the intended replacement is a small generator
-that walks the tree, encodes each namespace, and writes a `file_sd` target
-file; Prometheus reloads target files without a restart.
+Past that ceiling the scrape splits by namespace prefix, one job per prefix,
+with `/state.namespace_tree` generating the target list into a `file_sd` file:
 
     file_sd_configs:
       - files: ['/etc/prometheus/targets/namespaces.json']
         refresh_interval: 30s
 
-Namespaces churn as events start and end, and a relay may hold more than the
-per-scrape ceiling, so this becomes necessary rather than convenient once the
-relay carries production traffic.
+Label values use the moq-transport encoded form — tuple elements joined by
+`-`, other bytes as `.<hex>`:
+
+    moq-test/interop           ->  moq.2dtest-interop
+    conf.example.com / room 1  ->  conf.2eexample.2ecom-room.201
+
+Series exist only for live tracks: they disappear when a track ends and restart
+from zero if it returns, so counters need `increase()`/`rate()` rather than
+raw deltas across a track's lifetime.
+
+Counting is installed only when `admin.track_metrics_enabled` is true (default);
+with it false the endpoint returns 503.

docker/prometheus/prometheus.yml

@@ -73,33 +73,16 @@ scrape_configs:
       - target_label: __address__
         replacement: json-exporter:7979
 
-  # Per-track counters from the relay (openmoq/moqx#533). The namespace
-  # parameter is required, so each namespace is a target relabelled into the
-  # query. Values use the moq-transport encoded form: tuple elements joined
-  # by "-", other bytes as ".<hex>" (moq-test/interop -> moq.2dtest-interop).
-  #
-  # limit is a guard, not a top-N: a namespace holding more tracks than the
-  # limit is rejected with 400 rather than truncated, so the scrape fails
-  # loudly instead of graphing a set that reshuffles between scrapes.
-  #
-  # The list is static for now. /state.namespace_tree already enumerates live
-  # namespaces, so it should be generated into a file_sd target file and
-  # picked up without a reload — see docker/prometheus/README.md.
+  # Per-track counters from the relay (openmoq/moqx#533). Every parameter is
+  # optional, so one scrape covers all namespaces. limit is a guard, not a
+  # top-N: more live tracks than the limit returns 400 rather than a series
+  # set that reshuffles between scrapes.
   - job_name: moqx-track
     metrics_path: /metrics/track
     scrape_interval: 15s
     params:
       limit: ['1000']
     static_configs:
-      - targets:
-          - moq.2dtest-interop
-          - aiomoqt
-    relabel_configs:
-      - source_labels: [__address__]
-        target_label: __param_namespace
-      - source_labels: [__param_namespace]
-        target_label: ns_scrape
-      - target_label: __address__
-        replacement: moqx:8000
-      - target_label: instance
-        replacement: moqx-relay
+      - targets: ['moqx:8000']
+        labels:
+          instance: moqx-relay