feat(metrics): add more metrics around traffic

Author: XAMPPRocky

crates/xds/src/locality.rs

@@ -75,6 +75,15 @@ impl Locality {
         self.buffer.as_str().to_owned()
     }
 
+    /// The full colon separated locality, borrowed.
+    ///
+    /// Unlike [`Self::colon_separated_string`] this doesn't allocate, which
+    /// matters where it's used as a metric label on the packet path.
+    #[inline]
+    pub fn as_str(&self) -> &str {
+        self.buffer.as_str()
+    }
+
     #[inline]
     pub fn region(&self) -> &str {
         &self.buffer[..self.region]

docs/src/deployment/metrics.md

@@ -5,8 +5,9 @@ The following are metrics that Quilkin provides while in Proxy Mode.
 # ASN Maxmind Information
 
 If Quilkin is provided a a MaxmindDB GeoIP database, Quilkin will log the
-following information in the `maxmind information` log, as well as populate
-the following fields in any metrics with matching labels.
+following information in the `maxmind information` log. Only `country_code`, on
+`quilkin_session_active`, and `asn`, on the connection quality metrics below, are
+exported as labels; the rest are too high in cardinality to be.
 
 | Field           | Description                                   |
 |-----------------|-----------------------------------------------|
@@ -31,12 +32,24 @@ The proxy exposes the following general metrics:
         * `read`: when the proxy receives data from a downstream connection on the listening port.
         * `write`: when the proxy sends data to a downstream connection via the listening port.
 
-* `quilkin_packets_dropped_total{reason, asn, ip_prefix}` (Counter)
-
-  The total number of packets (not associated with any session) that were dropped by proxy.
-  Not that packets reflected by this metric were dropped at an earlier stage before they were associated with any session. For session based metrics, see the list of [session metrics][session-metrics] instead.
-    * `reason = NoConfiguredEndpoints`
-        * `NoConfiguredEndpoints`: No upstream endpoints were available to send the packet to. This can occur e.g if the endpoints cluster was scaled down to zero and the proxy is configured via a control plane.
+* `quilkin_packets_dropped_total{event, reason, filter, destination}` (Counter)
+
+  The total number of packets that were dropped by the proxy.
+    * The `reason` label is a closed set, so a breakdown built on it survives a
+      filter being renamed or an `errno` producing different text:
+        * `no_endpoint_match`: no upstream endpoint was available, or none matched the packet's routing token.
+        * `filter_drop`: a filter chose to drop the packet, ie the chain worked as configured.
+        * `filter_error`: a filter failed to process the packet.
+        * `socket_error`: the socket refused the packet, or the packet couldn't be built for it.
+        * `queue_full`: a send or receive queue was full.
+        * `invalid_packet`: the packet couldn't be parsed as a datagram Quilkin handles. The specific parse failure is logged rather than labelled.
+        * `session_limit`: the session limit was reached, so no session could be established.
+        * `internal`: Quilkin lost track of state it needed to forward the packet.
+    * The `filter` label is the filter responsible, and is empty when the drop
+      wasn't a filter's decision.
+    * The `destination` label is described under
+      [`quilkin_bytes_total`](#general-metrics), and is empty for packets dropped
+      before they were routed.
 
 * `quilkin_cluster_active`
 
@@ -47,27 +60,46 @@ The proxy exposes the following general metrics:
   The number of currently active upstream endpoints. Note that this tracks the number of endpoints that the proxy
   knows of rather than those that it is connected to (see [Session Metrics][session-metrics] instead for those)
 
-* `quilkin_bytes_total{event, asn, ip_prefix}`
+* `quilkin_bytes_total{event, destination}`
 
    The total number of bytes sent or received
   * The `event` label is either:
     * `read`: when the proxy receives data from a downstream connection on the listening port.
     * `write`: when the proxy sends data to a downstream connection via the listening port.
+  * The `destination` label is the cluster the packet was routed to, so traffic
+    can be attributed to a gameserver fleet rather than only counted in aggregate.
+    It carries the same values as `quilkin_active_endpoints`, so the two can be
+    joined.
+
+    The cluster comes from the routing decision itself, not from looking the
+    address up afterwards, so it is the cluster the packet actually went through
+    even when the same endpoint is configured in more than one. It is empty when
+    the destination didn't come from a cluster with a locality, which includes
+    destinations a filter decoded from the packet rather than selecting from the
+    cluster map. Note that `event=write` traffic carries the cluster of the
+    gameserver that sent it, not of the client it is going to.
 
-* `quilkin_packets_total{event, asn, ip_prefix}`
+* `quilkin_packets_total{event, destination}`
 
   The total number of packets sent or recieved.
-  * The `event` label is either:
-    * `read`: when the proxy receives data from a downstream connection on the listening port.
-    * `write`: when the proxy sends data to a downstream connection via the listening port.
+  * The labels are the same as [`quilkin_bytes_total`](#general-metrics).
+
+* `quilkin_packet_jitter{event}`
 
-* `quilkin_packet_jitter{event, asn, ip_prefix}`
+  The time between packets arriving at an I/O loop (in nanoseconds). This covers
+  every session the loop serves, so it is a whole-proxy figure; for the
+  distribution across players use `quilkin_session_jitter_seconds`.
 
-  The time between receiving new packets (in nanoseconds).
+  The series stops being exported when no packet arrived during an aggregation
+  interval, rather than continuing to publish the last value the proxy saw.
 
-* `quilkin_errors_total{event, asn, ip_prefix}`
+* `quilkin_errors_total{event, reason}`
 
   The total number of errors encountered while reading a packet from the upstream endpoint.
+  * The `reason` label is a closed set, replacing the previous free-text `display`
+    label. For I/O errors it names the `errno`, eg `invalid_input`,
+    `no_buffer_space` or `message_too_long`, rather than interpolating the
+    platform's error string.
 
 * `quilkin_game_traffic_tasks`
 
@@ -97,6 +129,88 @@ The proxy exposes the following metrics around sessions:
 
   The total number of sessions that have been created.
 
+* `quilkin_sessions_closed_total{reason}` (Counter)
+
+  The total number of sessions that have ended, by why they ended. A fall in
+  session count otherwise looks the same whether players left or their endpoints
+  vanished.
+  * `idle_timeout`: no traffic within the session TTL. UDP has no close, so this
+    is what a player leaving normally looks like.
+  * `endpoint_gone`: the endpoint the session was routed to is no longer in the
+    cluster map. Only reported for destinations that were in the cluster map when
+    the session was created, so an endpoint configured by name is never
+    misattributed here.
+  * `shutdown`: the proxy is shutting down.
+
+## Connection Quality Metrics
+
+A player's jitter and their ISP are per-player facts, so neither can be a metric
+label: concurrent sessions and the thousands of ASNs seen in a day of traffic both
+blow up cardinality. Instead the proxy tracks quality per session internally and
+periodically exports a projection whose series count is bounded by configuration
+rather than by traffic. See the `--service.udp.metrics.*` options for the
+tunables.
+
+Note the division of labour. Whether an individual *session* is having a bad time
+is a judgement the proxy makes, because it is the only thing holding that
+session's packet timing. Whether an *ISP* is having a bad time is left to the
+consumer of these metrics: a proxy carries on the order of a hundred concurrent
+sessions spread over thousands of ASNs, so no single one sees enough of any ASN to
+threshold on. The proxy exports the numerator and the denominator per ASN and
+expects them to be summed across the fleet before any conclusion is drawn.
+
+Interarrival jitter is measured against a monotonic clock, and gaps longer than a
+second are treated as the stream restarting rather than as jitter, so a player
+pausing doesn't register as a player with a bad connection.
+
+Sessions the proxy saw no traffic for during an interval are counted, but
+contribute no quality judgement.
+
+* `quilkin_session_jitter_seconds` (Histogram)
+
+  The distribution of per-session interarrival jitter of downstream packets, as
+  the RFC 3550 estimator. A histogram rather than a mean, because a mean of 0.5 ms
+  is compatible with a few percent of players at 80 ms, and those are the players
+  worth knowing about — `histogram_quantile` over this answers "what does the 99th
+  percentile player at this proxy see".
+
+  Only jitter is exported per session. Loss and round-trip time on the client's
+  leg need either packet sequence numbers or a client-side timestamp, and the
+  proxy has neither.
+
+* `quilkin_sessions_active_by_asn{asn}` (Gauge)
+
+  Active sessions by the client's ASN, for the largest
+  `--service.udp.metrics.top-asns` ASNs at this proxy, and the denominator for the
+  metric below.
+
+  Sessions belonging to any other ASN are counted under `asn="other"`, so the
+  breakdown always sums to the session total. Sessions whose client IP resolved to
+  no ASN are counted under `asn="unknown"`. Setting `--top-asns` to 0 disables
+  per-ASN reporting entirely.
+
+* `quilkin_client_sessions_degraded{asn, reason}` (Gauge)
+
+  The number of sessions breaching a quality threshold, by the client's ASN. Only
+  ASNs currently breaching are exported, so a healthy proxy publishes nothing
+  here and the series count follows the number of ISPs actually in trouble.
+  * `reason = jitter`: sessions at or above `--service.udp.metrics.jitter-threshold-ms`.
+
+  This is a count, not a verdict. For the affected share of an ISP, divide by
+  `quilkin_sessions_active_by_asn` — summing both across proxies first, since one
+  proxy's view of a single ASN is too small a sample to threshold on:
+
+  ```promql
+  sum by (asn) (quilkin_client_sessions_degraded{reason="jitter"})
+    / sum by (asn) (quilkin_sessions_active_by_asn)
+  ```
+
+* `quilkin_client_sessions_degraded_total{reason}` (Counter)
+
+  The total number of times a session was observed breaching a threshold, for
+  alerting on a rate without depending on a share the proxy would have to pick a
+  cut-off for.
+
 ## Filter Metrics
 Quilkin's filters use a set of generic metric keys, to make it easier to build visualisations that can account for
 a dynamic set of filters that can be added, removed, or updated at runtime with different configurations. All of

src/filters/chain.rs

@@ -296,8 +296,7 @@ impl Filter for FilterChain {
         // has rejected, and the destinations is empty, we passthrough to all.
         // Which mimics the old behaviour while avoid clones in most cases.
         if ctx.destinations.is_empty() {
-            ctx.destinations
-                .extend(ctx.endpoints.endpoints().into_iter().map(|ep| ep.address));
+            ctx.endpoints.destinations(ctx.destinations);
         }
 
         Ok(())
@@ -388,7 +387,18 @@ mod tests {
         config.filters.read(&mut context).unwrap();
         let expected = endpoints_fixture.clone();
 
-        assert_eq!(&*expected.endpoints(), &*context.destinations);
+        assert_eq!(
+            expected
+                .endpoints()
+                .into_iter()
+                .map(|ep| ep.address)
+                .collect::<Vec<_>>(),
+            context
+                .destinations
+                .iter()
+                .map(|d| d.address.clone())
+                .collect::<Vec<_>>()
+        );
         assert_eq!(
             "hello:odr:127.0.0.1:70",
             std::str::from_utf8(&context.contents).unwrap()
@@ -445,7 +455,14 @@ mod tests {
             (context.contents, context.metadata)
         };
         let expected = endpoints_fixture.clone();
-        assert_eq!(expected.endpoints(), dest);
+        assert_eq!(
+            expected
+                .endpoints()
+                .into_iter()
+                .map(|ep| ep.address)
+                .collect::<Vec<_>>(),
+            dest.iter().map(|d| d.address.clone()).collect::<Vec<_>>()
+        );
         assert_eq!(b"hello:odr:127.0.0.1:70:odr:127.0.0.1:70", &*contents);
         assert_eq!(
             "receive:receive",

src/filters/decryptor.rs

@@ -109,8 +109,12 @@ impl Filter for Decryptor {
                         };
 
                         self.decode_chacha20(nonce, edata);
-                        ctx.destinations
-                            .push(Self::decode_destination(edata).into());
+                        // The address came out of the packet, not out of the
+                        // cluster map, so there is no cluster it was routed from
+                        ctx.destinations.push(crate::net::Destination::new(
+                            Self::decode_destination(edata).into(),
+                            None,
+                        ));
                         Ok(())
                     }
                 }
@@ -286,7 +290,12 @@ mod tests {
             filter.read(&mut ctx).unwrap();
             assert_eq!(
                 std::net::SocketAddr::from(expected),
-                ctx.destinations.pop().unwrap().to_socket_addr().unwrap()
+                ctx.destinations
+                    .pop()
+                    .unwrap()
+                    .address
+                    .to_socket_addr()
+                    .unwrap()
             );
         }
 
@@ -312,7 +321,12 @@ mod tests {
             filter.read(&mut ctx).unwrap();
             assert_eq!(
                 std::net::SocketAddr::from(expected),
-                ctx.destinations.pop().unwrap().to_socket_addr().unwrap()
+                ctx.destinations
+                    .pop()
+                    .unwrap()
+                    .address
+                    .to_socket_addr()
+                    .unwrap()
             );
         }
     }

src/filters/error.rs

@@ -48,6 +48,44 @@ impl FilterError {
             Self::Custom(custom) => custom,
         }
     }
+
+    /// The bounded drop reason this error corresponds to.
+    ///
+    /// A filter dropping a packet by design is separated from a filter failing:
+    /// the first is the chain working as configured, the second is a fault.
+    #[inline]
+    pub fn drop_reason(&self) -> crate::metrics::DropReason {
+        use crate::metrics::DropReason;
+
+        match self {
+            Self::FirewallDenied | Self::Dropped | Self::RateLimitExceeded => {
+                DropReason::FilterDrop
+            }
+            Self::TokenRouter(tr) => tr.drop_reason(),
+            Self::NoValueCaptured | Self::MatchNoMetadata | Self::Custom(_) => {
+                DropReason::FilterError
+            }
+            Self::Io(..) => DropReason::SocketError,
+        }
+    }
+
+    /// The filter that produced this error, empty when it didn't come from a
+    /// specific filter.
+    ///
+    /// Kept apart from [`Self::drop_reason`] so renaming or reconfiguring a
+    /// filter doesn't change the reason vocabulary.
+    #[inline]
+    pub fn filter_name(&self) -> &'static str {
+        match self {
+            Self::NoValueCaptured => "capture",
+            Self::TokenRouter(..) => "token_router",
+            Self::FirewallDenied => "firewall",
+            Self::MatchNoMetadata => "match",
+            Self::Dropped => "drop",
+            Self::RateLimitExceeded => "rate_limit",
+            Self::Io(..) | Self::Custom(..) => "",
+        }
+    }
 }
 
 impl std::error::Error for FilterError {}

src/filters/load_balancer.rs

@@ -86,7 +86,7 @@ mod tests {
             filter.read(&mut context).unwrap();
         }
 
-        dest
+        dest.into_iter().map(|d| d.address).collect()
     }
 
     #[tokio::test]