prometheus: per-namespace track scrape, non-overlapping targets

/metrics/track matches a namespace prefix, so a parent and its child return
the same tracks and every aggregate double-counts. Targets are top-level
namespaces only: they do not overlap and their prefixes still cover
everything beneath them.

The endpoint documents itself as enabled by default but the resolved value
was false, so it shipped dead; the deployed config states it explicitly.

Author: gmarzot

.gitignore

@@ -43,3 +43,7 @@ docker/grafana/provisioning/dashboards/archive/
 
 # Version stamp for source trees with no .git (cmake/MoqxVersion.cmake).
 /VERSION
+
+# Python bytecode from local tooling.
+__pycache__/
+*.pyc

docker/config.docker.yaml

@@ -59,5 +59,6 @@ services:
 
 admin:
   port: ${MOQX_ADMIN_PORT}
+  track_metrics_enabled: true
   address: "${MOQX_BIND_ADDR}"
   plaintext: true

docker/prometheus/README.md

@@ -22,8 +22,11 @@ read as "no live tracks".
 Namespaces come and go with events, so the target list cannot be static.
 `namespace-targets.py` walks the relay's `/state` namespace tree every 30s and
 writes one target per namespace to a file Prometheus rereads without a restart.
-It collects every node carrying a namespace rather than only the leaves, since
-tracks can be published at any depth.
+It emits only top-level namespaces. The endpoint matches a prefix, so scraping
+a parent and its child would return the same tracks twice and every aggregate
+would double-count; top-level namespaces do not overlap, and their prefixes
+still cover every track beneath them. If one grows past the limit it has to be
+split by descending a level.
 
 Targets are namespace values in the moq-transport safe form — `[A-Za-z0-9_]`
 passes through, every other byte becomes `.<hex>`, tuple elements join with

docker/prometheus/__pycache__/namespace-targets.cpython-310.pyc

@@ -38,14 +38,20 @@ def safe_namespace(tuple_elements):
     return "-".join(safe_element(e) for e in tuple_elements)
 
 
-def walk(node, found):
-    """Collect every node carrying a namespace, not just the leaves: tracks can
-    be published at any depth."""
-    full = node.get("full_namespace") or []
-    if full:
-        found.append(full)
-    for child in (node.get("children") or {}).values():
-        walk(child, found)
+def top_level(tree):
+    """Only the tree's immediate children.
+
+    The endpoint matches a namespace prefix, so scraping both a parent and its
+    child returns the same tracks twice and every aggregate double-counts.
+    Top-level namespaces do not overlap each other and their prefixes still
+    cover every track beneath them.
+    """
+    out = []
+    for child in (tree.get("children") or {}).values():
+        full = child.get("full_namespace") or []
+        if full:
+            out.append(full)
+    return out
 
 
 def namespaces():
@@ -55,7 +61,7 @@ def namespaces():
     for service in (state.get("services") or {}).values():
         tree = service.get("namespace_tree")
         if tree:
-            walk(tree, found)
+            found.extend(top_level(tree))
     # A namespace can appear under more than one service.
     return sorted({tuple(ns) for ns in found})
 

docker/prometheus/namespace-targets.py

@@ -227,7 +227,7 @@ struct ServiceConfig {
 struct AdminConfig {
   folly::SocketAddress address;
   std::optional<TlsConfig> tls;
-  bool trackMetricsEnabled{false};
+  bool trackMetricsEnabled{true};
   uint32_t trackMetricsLimit{10};
   uint32_t trackMetricsMaxLimit{1000};
 };