Skip to content

Add metrics for messages sent to/received from clients #329

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alexbozhenko wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add metrics for messages sent to/received from clients #329

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
e5d80be
add new metrics
alexbozhenko Feb 25, 2026
4413fd4
update dashboard to use the new metric
alexbozhenko Feb 25, 2026
6389857
fix autoformat
alexbozhenko Feb 25, 2026
f07c836
clarify help text of old graphs
alexbozhenko Feb 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
70 changes: 39 additions & 31 deletions docker-compose/grafana/provisioning/dashboards/nats-surveyor-dashboard.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
"liveNow": false,
"panels": [
{
"description": "Combined messages received by all servers",
"description": "Aggregate messages received by all servers from clients, routes, gateways and leafnodes",
"fieldConfig": {
"defaults": {
"color": {
Expand Down Expand Up @@ -101,7 +101,7 @@
}
},
{
"description": "Combined messages sent by all servers",
"description": "Aggregate messages sent by all servers to clients, routes, gateways and leafnodes",
"fieldConfig": {
"defaults": {
"color": {
Expand Down Expand Up @@ -175,7 +175,7 @@
}
},
{
"description": "Combined bytes received by all servers",
"description": "Aggregate bytes received by all servers from clients, routes, gateways and leafnodes",
"fieldConfig": {
"defaults": {
"color": {
Expand Down Expand Up @@ -249,7 +249,7 @@
}
},
{
"description": "Combined bytes sent by all servers",
"description": "Aggregate bytes sent by all servers to clients, routes, gateways and leafnodes",
"fieldConfig": {
"defaults": {
"color": {
Expand Down Expand Up @@ -2040,7 +2040,7 @@
}
},
{
"description": "Combined total messages for all servers",
"description": "Aggregate messages across all servers and all connection types: clients, routes, gateways and leafnodes",
"fieldConfig": {
"defaults": {
"color": {
Expand Down Expand Up @@ -2136,7 +2136,7 @@
}
},
{
"description": "Combined cluster network usage",
"description": "Aggregate network usage across all servers and all connection types: clients, routes, gateways and leafnodes",
"fieldConfig": {
"defaults": {
"color": {
Expand Down Expand Up @@ -3731,7 +3731,7 @@
"id": 24,
"panels": [
{
"description": "Data sent per second",
"description": "Data sent to clients per second",
"fieldConfig": {
"defaults": {
"color": {
Expand Down Expand Up @@ -3785,7 +3785,7 @@
"overrides": []
},
"gridPos": {
"h": 7,
"h": 9,
"w": 12,
"x": 0,
"y": 9
Expand All @@ -3801,7 +3801,9 @@
"min"
],
"displayMode": "table",
"placement": "bottom"
"placement": "bottom",
"sortBy": "Name",
"sortDesc": false
},
"tooltip": {
"mode": "single"
Expand All @@ -3810,16 +3812,16 @@
"pluginVersion": "8.3.4",
"targets": [
{
"expr": "rate(nats_core_sent_bytes{server_cluster=~\"$cluster\"}[1m])",
"expr": "rate(nats_core_sent_client_bytes{server_cluster=~\"$cluster\"}[1m])",
"legendFormat": "{{server_name}}",
"refId": "A"
}
],
"title": "Bytes Sent",
"title": "Bytes Sent to Clients",
"type": "timeseries"
},
{
"description": "Data received per second",
"description": "Data received from clients per second",
"fieldConfig": {
"defaults": {
"color": {
Expand Down Expand Up @@ -3873,7 +3875,7 @@
"overrides": []
},
"gridPos": {
"h": 7,
"h": 9,
"w": 12,
"x": 12,
"y": 9
Expand All @@ -3889,7 +3891,9 @@
"min"
],
"displayMode": "table",
"placement": "bottom"
"placement": "bottom",
"sortBy": "Name",
"sortDesc": false
},
"tooltip": {
"mode": "single"
Expand All @@ -3898,23 +3902,23 @@
"pluginVersion": "8.3.4",
"targets": [
{
"expr": "rate(nats_core_recv_bytes{server_cluster=~\"$cluster\"}[1m])",
"expr": "rate(nats_core_recv_client_bytes{server_cluster=~\"$cluster\"}[1m])",
"legendFormat": "{{server_name}}",
"refId": "A"
}
],
"title": "Bytes Received",
"title": "Bytes Received from Clients",
"type": "timeseries"
},
{
"description": "Messages sent per second",
"description": "Messages sent to clients per second",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisLabel": "msgs/s",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
Expand Down Expand Up @@ -3961,10 +3965,10 @@
"overrides": []
},
"gridPos": {
"h": 7,
"h": 9,
"w": 12,
"x": 0,
"y": 16
"y": 18
},
"id": 68,
"interval": "",
Expand All @@ -3977,7 +3981,9 @@
"min"
],
"displayMode": "table",
"placement": "bottom"
"placement": "bottom",
"sortBy": "Name",
"sortDesc": false
},
"tooltip": {
"mode": "single"
Expand All @@ -3986,23 +3992,23 @@
"pluginVersion": "8.3.4",
"targets": [
{
"expr": "rate(nats_core_sent_msgs_count{server_cluster=~\"$cluster\"}[1m])",
"expr": "rate(nats_core_sent_client_msgs_count{server_cluster=~\"$cluster\"}[1m])",
"legendFormat": "{{server_name}}",
"refId": "A"
}
],
"title": "Messages Sent",
"title": "Messages Sent to Clients",
"type": "timeseries"
},
{
"description": "Messages received per second",
"description": "Messages received from clients per second",
"fieldConfig": {
"defaults": {
"color": {
"mode": "palette-classic"
},
"custom": {
"axisLabel": "",
"axisLabel": "msgs/s",
"axisPlacement": "auto",
"barAlignment": 0,
"drawStyle": "line",
Expand Down Expand Up @@ -4049,10 +4055,10 @@
"overrides": []
},
"gridPos": {
"h": 7,
"h": 9,
"w": 12,
"x": 12,
"y": 16
"y": 18
},
"id": 69,
"interval": "",
Expand All @@ -4065,7 +4071,9 @@
"min"
],
"displayMode": "table",
"placement": "bottom"
"placement": "bottom",
"sortBy": "Name",
"sortDesc": false
},
"tooltip": {
"mode": "single"
Expand All @@ -4074,12 +4082,12 @@
"pluginVersion": "8.3.4",
"targets": [
{
"expr": "rate(nats_core_recv_msgs_count{server_cluster=~\"$cluster\"}[1m])",
"expr": "rate(nats_core_recv_client_msgs_count{server_cluster=~\"$cluster\"}[1m])",
"legendFormat": "{{server_name}}",
"refId": "A"
}
],
"title": "Messages Received",
"title": "Messages Received from Clients",
"type": "timeseries"
}
],
Expand Down Expand Up @@ -4959,4 +4967,4 @@
"uid": "GGxJ_5oZy",
"version": 1,
"weekStart": ""
}
}
24 changes: 20 additions & 4 deletions surveyor/collector_statz.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,8 +59,12 @@
NumSubs *GaugeVec
SentMsgs *CounterVec
SentBytes *CounterVec
SentClientMsgs *CounterVec
SentClientBytes *CounterVec
RecvMsgs *CounterVec
RecvBytes *CounterVec
RecvClientMsgs *CounterVec
RecvClientBytes *CounterVec
SlowConsumers *GaugeVec
RTT *GaugeVec
Routes *GaugeVec
Expand Down Expand Up @@ -382,10 +386,14 @@
sc.descs.TotalConnections = newCounterVec(newName("total_connection_count"), "Total number of client connections serviced counter", sc.constLabels, sc.serverLabels)
sc.descs.ActiveAccounts = newGaugeVec(newName("active_account_count"), "Number of active accounts gauge", sc.constLabels, sc.serverLabels)
sc.descs.NumSubs = newGaugeVec(newName("subs_count"), "Current number of subscriptions gauge", sc.constLabels, sc.serverLabels)
sc.descs.SentMsgs = newCounterVec(newName("sent_msgs_count"), "Number of messages sent counter", sc.constLabels, sc.serverLabels)
sc.descs.SentBytes = newCounterVec(newName("sent_bytes"), "Number of bytes sent counter", sc.constLabels, sc.serverLabels)
sc.descs.RecvMsgs = newCounterVec(newName("recv_msgs_count"), "Number of messages received counter", sc.constLabels, sc.serverLabels)
sc.descs.RecvBytes = newCounterVec(newName("recv_bytes"), "Number of bytes received counter", sc.constLabels, sc.serverLabels)
sc.descs.SentMsgs = newCounterVec(newName("sent_msgs_count"), "Number of messages sent by the server to all connections including clients, routes, gateways and leafnodes counter", sc.constLabels, sc.serverLabels)
sc.descs.SentBytes = newCounterVec(newName("sent_bytes"), "Number of bytes sent by the server to all connections including clients, routes, gateways and leafnodes counter", sc.constLabels, sc.serverLabels)
sc.descs.SentClientMsgs = newCounterVec(newName("sent_client_msgs_count"), "Number of messages sent by the server to clients only counter", sc.constLabels, sc.serverLabels)
sc.descs.SentClientBytes = newCounterVec(newName("sent_client_bytes"), "Number of bytes sent by the server to clients only counter", sc.constLabels, sc.serverLabels)
sc.descs.RecvMsgs = newCounterVec(newName("recv_msgs_count"), "Number of messages received by the server from all connections including clients, routes, gateways and leafnodes counter", sc.constLabels, sc.serverLabels)
sc.descs.RecvBytes = newCounterVec(newName("recv_bytes"), "Number of bytes received by the server from all connections including clients, routes, gateways and leafnodes counter", sc.constLabels, sc.serverLabels)
sc.descs.RecvClientMsgs = newCounterVec(newName("recv_client_msgs_count"), "Number of messages received by the server from clients only counter", sc.constLabels, sc.serverLabels)
sc.descs.RecvClientBytes = newCounterVec(newName("recv_client_bytes"), "Number of bytes received by the server from clients only counter", sc.constLabels, sc.serverLabels)
sc.descs.SlowConsumers = newGaugeVec(newName("slow_consumer_count"), "Number of slow consumers gauge", sc.constLabels, sc.serverLabels)
sc.descs.RTT = newGaugeVec(newName("rtt_nanoseconds"), "RTT in nanoseconds gauge", sc.constLabels, sc.serverLabels)
sc.descs.Routes = newGaugeVec(newName("route_count"), "Number of active routes gauge", sc.constLabels, sc.serverLabels)
Expand Down Expand Up @@ -1411,8 +1419,12 @@
sc.descs.NumSubs,
sc.descs.SentMsgs,
sc.descs.SentBytes,
sc.descs.SentClientMsgs,
sc.descs.SentClientBytes,
sc.descs.RecvMsgs,
sc.descs.RecvBytes,
sc.descs.RecvClientMsgs,
sc.descs.RecvClientBytes,
sc.descs.SlowConsumers,
sc.descs.RTT,
sc.descs.Routes,
Expand Down Expand Up @@ -1651,8 +1663,12 @@
metrics.newGaugeMetric(sc.descs.NumSubs, float64(sm.Stats.NumSubs), labels)
metrics.newCounterMetric(sc.descs.SentMsgs, float64(sm.Stats.Sent.Msgs), labels)
metrics.newCounterMetric(sc.descs.SentBytes, float64(sm.Stats.Sent.Bytes), labels)
metrics.newCounterMetric(sc.descs.SentClientMsgs, float64(sm.Stats.SentToClients.Msgs), labels)

Check failure on line 1666 in surveyor/collector_statz.go

View workflow job for this annotation

GitHub Actions / test (1.25, macOS-latest) 

sm.Stats.SentToClients undefined (type server.ServerStats has no field or method SentToClients)

Check failure on line 1666 in surveyor/collector_statz.go

View workflow job for this annotation

GitHub Actions / test (1.25, ubuntu-latest) 

sm.Stats.SentToClients undefined (type server.ServerStats has no field or method SentToClients)
metrics.newCounterMetric(sc.descs.SentClientBytes, float64(sm.Stats.SentToClients.Bytes), labels)

Check failure on line 1667 in surveyor/collector_statz.go

View workflow job for this annotation

GitHub Actions / test (1.25, macOS-latest) 

sm.Stats.SentToClients undefined (type server.ServerStats has no field or method SentToClients)

Check failure on line 1667 in surveyor/collector_statz.go

View workflow job for this annotation

GitHub Actions / test (1.25, ubuntu-latest) 

sm.Stats.SentToClients undefined (type server.ServerStats has no field or method SentToClients)
metrics.newCounterMetric(sc.descs.RecvMsgs, float64(sm.Stats.Received.Msgs), labels)
metrics.newCounterMetric(sc.descs.RecvBytes, float64(sm.Stats.Received.Bytes), labels)
metrics.newCounterMetric(sc.descs.RecvClientMsgs, float64(sm.Stats.ReceivedFromClients.Msgs), labels)

Check failure on line 1670 in surveyor/collector_statz.go

View workflow job for this annotation

GitHub Actions / test (1.25, macOS-latest) 

sm.Stats.ReceivedFromClients undefined (type server.ServerStats has no field or method ReceivedFromClients)

Check failure on line 1670 in surveyor/collector_statz.go

View workflow job for this annotation

GitHub Actions / test (1.25, ubuntu-latest) 

sm.Stats.ReceivedFromClients undefined (type server.ServerStats has no field or method ReceivedFromClients)
metrics.newCounterMetric(sc.descs.RecvClientBytes, float64(sm.Stats.ReceivedFromClients.Bytes), labels)

Check failure on line 1671 in surveyor/collector_statz.go

View workflow job for this annotation

GitHub Actions / test (1.25, macOS-latest) 

sm.Stats.ReceivedFromClients undefined (type server.ServerStats has no field or method ReceivedFromClients)

Check failure on line 1671 in surveyor/collector_statz.go

View workflow job for this annotation

GitHub Actions / test (1.25, ubuntu-latest) 

sm.Stats.ReceivedFromClients undefined (type server.ServerStats has no field or method ReceivedFromClients)
metrics.newGaugeMetric(sc.descs.SlowConsumers, float64(sm.Stats.SlowConsumers), labels)
metrics.newGaugeMetric(sc.descs.RTT, float64(sc.rtts[sm.Server.ID]), labels)
metrics.newGaugeMetric(sc.descs.Routes, float64(len(sm.Stats.Routes)), labels)
Expand Down
Loading