Skip to content

Commit 81ff2b8

Browse files
nubtronbrett0000FF
nubtron
and
brett0000FF
authored
Implement Kuma dashboards and monitors (DataDog#20611)
* Implement Kuma dashboards and monitors. * Tweak control plane dashboard. * Tweak the service communication dashboard. * Fix latency query. * Shorten monitor message. * Shorten monitor description. * Commit suggestion Co-authored-by: Brett Blue <84536271+brett0000FF@users.noreply.github.com> * Commit suggestion Co-authored-by: Brett Blue <84536271+brett0000FF@users.noreply.github.com> * Fix typo Co-authored-by: Brett Blue <84536271+brett0000FF@users.noreply.github.com> * Use cluster_name as template variable and apply to all widgets. * Remove the "Dataplane Server" group and edit layout to preserve symmetry. * Fix title. * Fix "Request Volume" widget title * Add note about template variables to service communication dashboard. * Tweak layout * Change color from red to blue to make it less alarming. * Use "rate" properly. --------- Co-authored-by: Brett Blue <84536271+brett0000FF@users.noreply.github.com>
1 parent 5fbc09e commit 81ff2b8

8 files changed

Lines changed: 2551 additions & 0 deletions

‎kuma/assets/dashboards/kuma_control_plane.json‎

Lines changed: 1628 additions & 0 deletions
Large diffs are not rendered by default.

‎kuma/assets/dashboards/kuma_service_communication.json‎

Lines changed: 743 additions & 0 deletions
Large diffs are not rendered by default.

‎kuma/assets/monitors/api_server_error_rate.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"version": 2,
3+
"created_at": "2025-06-30",
4+
"last_updated_at": "2025-06-30",
5+
"title": "Kuma API Server High Error Rate",
6+
"tags": [
7+
"integration:kuma"
8+
],
9+
"description": "The Kuma API Server exposes REST APIs for configuration management. This monitor alerts when the error rate (4xx and 5xx responses) exceeds 10% of total requests, indicating potential issues with API operations or client requests.",
10+
"definition": {
11+
"message": "{{#is_alert}}\n\nKuma API Server is experiencing a high error rate on {{instance_id.name}} in zone {{zone.name}}.\n\nError rate: {{value}}%\n\nThis indicates that more than 10% of API requests are returning 4xx or 5xx status codes. Check the API server logs for details about the failing requests.\n\nCommon causes:\n- Invalid API requests (4xx errors)\n- Internal server errors (5xx errors)\n- Authentication/authorization issues\n- Resource constraints\n\n{{/is_alert}}",
12+
"name": "[Kuma] API Server High Error Rate",
13+
"options": {
14+
"thresholds": {
15+
"critical": 10,
16+
"warning": 5
17+
},
18+
"notify_audit": false,
19+
"include_tags": true,
20+
"new_group_delay": 60,
21+
"silenced": {},
22+
"notify_no_data": false,
23+
"renotify_interval": 0,
24+
"require_full_window": false,
25+
"escalation_message": "The Kuma API server error rate remains high. This may impact configuration updates and management operations."
26+
},
27+
"priority": 2,
28+
"query": "sum(last_10m):( sum:kuma.api_server.http_request_duration_seconds.count{code_class:4xx OR code_class:5xx} by {instance_id,zone}.as_count() / sum:kuma.api_server.http_request_duration_seconds.count{*} by {instance_id,zone}.as_count() ) * 100 > 10",
29+
"tags": [
30+
"integration:kuma"
31+
],
32+
"type": "query alert"
33+
}
34+
}

‎kuma/assets/monitors/multiple_leaders_detected.json‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
{
2+
"version": 2,
3+
"created_at": "2025-06-30",
4+
"last_updated_at": "2025-06-30",
5+
"title": "Kuma Multiple Leaders Detected",
6+
"tags": [
7+
"integration:kuma"
8+
],
9+
"description": "Each Kuma zone must have exactly one leader instance for proper operation. This monitor alerts when multiple leaders are detected, which can cause split-brain scenarios and conflicting policy decisions.",
10+
"definition": {
11+
"message": "{{#is_alert}}\n\nMultiple Kuma leaders detected in zone {{zone.name}}.\n\nLeader count: {{value}}\n\nMultiple leaders can cause split-brain scenarios.{{/is_alert}}",
12+
"name": "[Kuma] Multiple Leaders Detected",
13+
"options": {
14+
"thresholds": {
15+
"critical": 1.5
16+
},
17+
"notify_audit": false,
18+
"include_tags": true,
19+
"new_group_delay": 60,
20+
"silenced": {},
21+
"notify_no_data": false,
22+
"renotify_interval": 300,
23+
"require_full_window": true,
24+
"evaluation_delay": 30
25+
},
26+
"priority": 1,
27+
"query": "max(last_5m):sum:kuma.leader{*} by {zone,cluster_name} > 1.5",
28+
"tags": [
29+
"integration:kuma"
30+
],
31+
"type": "query alert"
32+
}
33+
}

‎kuma/assets/monitors/no_leader_detected.json‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
{
2+
"version": 2,
3+
"created_at": "2025-06-30",
4+
"last_updated_at": "2025-06-30",
5+
"title": "Kuma No Leader Detected",
6+
"tags": [
7+
"integration:kuma"
8+
],
9+
"description": "Each Kuma zone must have exactly one leader instance for proper operation. This monitor alerts when no leader is detected, which prevents policy updates and proper control plane operation.",
10+
"definition": {
11+
"message": "{{#is_alert}}\n\nNo Kuma leader detected in zone {{zone.name}}.\n\nLeader count: {{value}}{{/is_alert}}",
12+
"name": "[Kuma] No Leader Detected",
13+
"options": {
14+
"thresholds": {
15+
"critical": 0.5
16+
},
17+
"notify_audit": false,
18+
"include_tags": true,
19+
"new_group_delay": 300,
20+
"silenced": {},
21+
"notify_no_data": true,
22+
"no_data_timeframe": 20,
23+
"renotify_interval": 300,
24+
"require_full_window": true,
25+
"evaluation_delay": 60
26+
},
27+
"priority": 1,
28+
"query": "min(last_10m):sum:kuma.leader{*} by {zone,cluster_name} < 0.5",
29+
"tags": [
30+
"integration:kuma"
31+
],
32+
"type": "query alert"
33+
}
34+
}

‎kuma/assets/monitors/xds_delivery_latency.json‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
{
2+
"version": 2,
3+
"created_at": "2025-06-30",
4+
"last_updated_at": "2025-06-30",
5+
"title": "Kuma xDS High Configuration Delivery Latency",
6+
"tags": [
7+
"integration:kuma"
8+
],
9+
"description": "Monitors the 99th percentile latency of xDS configuration delivery to data plane proxies. High latency can delay policy updates and impact the responsiveness of configuration changes in your service mesh.",
10+
"definition": {
11+
"message": "{{#is_alert}}\n\nKuma xDS configuration delivery is experiencing high latency on {{instance_id.name}} in zone {{zone.name}}.\n\nP99 latency: {{value}}ms\n\nHigh configuration delivery latency delays the application of policy updates to data plane proxies. This can impact:\n- Time to apply traffic routing changes\n- Security policy enforcement delays\n- Service discovery update lag\n\nPossible causes:\n- Control plane resource constraints\n- Large number of data plane connections\n- Complex policy configurations\n- Network latency between control and data planes\n\n{{/is_alert}}",
12+
"name": "[Kuma] xDS High Configuration Delivery Latency",
13+
"options": {
14+
"thresholds": {
15+
"critical": 5000,
16+
"warning": 2000
17+
},
18+
"notify_audit": false,
19+
"include_tags": true,
20+
"new_group_delay": 60,
21+
"silenced": {},
22+
"notify_no_data": false,
23+
"renotify_interval": 0,
24+
"require_full_window": false
25+
},
26+
"priority": 3,
27+
"query": "avg(last_10m):avg:kuma.xds.delivery.quantile{quantile:0.99} by {instance_id,zone} > 5000",
28+
"tags": [
29+
"integration:kuma"
30+
],
31+
"type": "query alert"
32+
}
33+
}

‎kuma/assets/monitors/xds_generation_errors.json‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
{
2+
"version": 2,
3+
"created_at": "2025-06-30",
4+
"last_updated_at": "2025-06-30",
5+
"title": "Kuma xDS Configuration Generation Errors",
6+
"tags": [
7+
"integration:kuma"
8+
],
9+
"description": "The xDS protocol enables dynamic configuration updates from the control plane to data plane proxies. This monitor alerts when configuration generation errors occur, which can prevent policy updates from reaching the data plane.",
10+
"definition": {
11+
"message": "{{#is_alert}}\n\nKuma is experiencing xDS configuration generation errors on {{instance_id.name}} in zone {{zone.name}}.\n\nError rate: {{value}} errors per second\n\nConfiguration generation errors prevent policy updates from being applied to data plane proxies. This can lead to:\n- Outdated routing rules\n- Missing security policies \n- Incorrect load balancing configurations\n\nCheck the control plane logs for detailed error messages.\n\n{{/is_alert}}",
12+
"name": "[Kuma] xDS Configuration Generation Errors",
13+
"options": {
14+
"thresholds": {
15+
"critical": 5,
16+
"warning": 1
17+
},
18+
"notify_audit": false,
19+
"include_tags": true,
20+
"new_group_delay": 60,
21+
"silenced": {},
22+
"notify_no_data": false,
23+
"renotify_interval": 300,
24+
"require_full_window": false,
25+
"timeout_h": 0,
26+
"evaluation_delay": 0
27+
},
28+
"priority": 2,
29+
"query": "avg(last_5m):sum:kuma.xds.generation_errors.count{*} by {instance_id,zone}.as_rate() > 5",
30+
"tags": [
31+
"integration:kuma"
32+
],
33+
"type": "query alert"
34+
}
35+
}

‎kuma/manifest.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -46,6 +46,17 @@
4646
"kuma-cp",
4747
"kuma-dp"
4848
]
49+
},
50+
"dashboards": {
51+
"Kuma Control Plane": "assets/dashboards/kuma_control_plane.json",
52+
"Kuma Service Communication": "assets/dashboards/kuma_service_communication.json"
53+
},
54+
"monitors": {
55+
"Kuma No Leader Detected": "assets/monitors/no_leader_detected.json",
56+
"Kuma Multiple Leaders Detected": "assets/monitors/multiple_leaders_detected.json",
57+
"Kuma API Server High Error Rate": "assets/monitors/api_server_error_rate.json",
58+
"Kuma xDS Configuration Generation Errors": "assets/monitors/xds_generation_errors.json",
59+
"Kuma xDS High Configuration Delivery Latency": "assets/monitors/xds_delivery_latency.json"
4960
}
5061
},
5162
"author": {

0 commit comments

Comments
 (0)