Skip to content

Commit eea576f

Browse files
nubtronnubtron
authored
Kuma Logs (DataDog#20597)
* Add assets for Kuma logs. * Update Kuma logs. * Set app IDs. * Remove extra line from test. * Add logs to manifest.json. * Add saved views. * Add test results. * Reformat tests yaml * Attempt fixing yaml parsing issues. * Improve formatting. * Set "kuma" as the pipeline source. * Add standard attributes to facets. * Remove kuma-dp source from saved view. * Remove columns from saved view. * Remove the service remapper - it doesn't seem to be needed. * Update support and parsing rules: * Avoid using negated classes in regexes when possible (except for urls, for which it doesn't seem to be standard practice) * Use rules with alternatives such as "(-|<ITEM>)" or "(unknown|<ITEM>)" to discard placeholder values for empty fields. * Delete unused support rules * Restore the service remapper. * Update tests. * Add authority helper rule. * Update test results * Remove dash for trace id. * Go back to negated class for kuma.upstream.host since I'm not certain about the format. * Use "notOpeningParens" for kuma_dp_tcp_log upstream host. * Remove service remapper.
1 parent e7589a6 commit eea576f

4 files changed

Lines changed: 348 additions & 0 deletions

File tree

‎kuma/assets/logs/kuma.yaml‎

Lines changed: 160 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,160 @@
1+
id: kuma
2+
metric_id: kuma
3+
backend_only: false
4+
facets:
5+
- groups:
6+
- Web Access
7+
name: Browser
8+
path: http.useragent_details.browser.family
9+
source: log
10+
- groups:
11+
- Web Access
12+
name: Device
13+
path: http.useragent_details.device.family
14+
source: log
15+
- groups:
16+
- Web Access
17+
name: OS
18+
path: http.useragent_details.os.family
19+
source: log
20+
- description: Destination Kuma service for the request.
21+
facetType: list
22+
groups:
23+
- Kuma
24+
name: kuma.destination_service
25+
path: kuma.destination_service
26+
source: log
27+
type: string
28+
- description: Name of the Kuma mesh.
29+
facetType: list
30+
groups:
31+
- Kuma
32+
name: kuma.mesh
33+
path: kuma.mesh
34+
source: log
35+
type: string
36+
- description: Source Kuma service for the request (without port).
37+
facetType: list
38+
groups:
39+
- Kuma
40+
name: kuma.source_address_without_port
41+
path: kuma.source_address_without_port
42+
source: log
43+
type: string
44+
- description: Source Kuma service for the request.
45+
facetType: list
46+
groups:
47+
- Kuma
48+
name: kuma.source_service
49+
path: kuma.source_service
50+
source: log
51+
type: string
52+
- description: Upstream host for the request.
53+
facetType: list
54+
groups:
55+
- Kuma
56+
name: kuma.upstream.host
57+
path: kuma.upstream.host
58+
source: log
59+
type: string
60+
pipeline:
61+
type: pipeline
62+
name: Kuma
63+
enabled: true
64+
filter:
65+
query: source:kuma
66+
processors:
67+
- type: grok-parser
68+
name: Kuma Log Parser
69+
enabled: true
70+
source: message
71+
samples:
72+
- '[2025-06-26T21:29:21.535Z] - default 10.42.1.5(unknown)->10.42.1.5:6379(redis_kuma-demo_svc_6379) took 14004ms, sent 6195 bytes, received: 194 bytes'
73+
- '[2025-06-26T13:19:18.314Z] default "GET /api?timeout=32s HTTP/1.1" 404 NR 0 0 0 - "10.42.1.26" "kubectl/v1.31.5+k3s1 (linux/arm64) kubernetes/56ec5dd" "-" "0efbfa26-1e2d-9144-919a-e9985d0eab30" "localhost:8080" "unknown" "edge-gateway_kuma-demo_svc" "10.42.3.27" "-"'
74+
75+
grok:
76+
supportRules: |-
77+
isoTimestamp %{date("yyyy-MM-dd'T'HH:mm:ss.SSSZZ")}
78+
goTimestamp %{date("yyyy/MM/dd HH:mm:ss")}
79+
notTab %{regex("[^\\t]*")}
80+
notQuote %{regex("[^\"]*")}
81+
notOpeningParens %{regex("[^\\(]*")}
82+
serviceName %{regex("[a-zA-Z0-9_\\*-]*")}
83+
loggerName %{regex("[a-zA-Z0-9_\\.-]+")}
84+
xForwardedFor %{regex("[0-9\\., ]+")}
85+
httpVersion %{regex("HTTP/[0-9\\.]+")}
86+
traceId %{regex("[0-9a-f]+")}
87+
authority %{regex("[0-9a-zA-Z@:\\._-]+")}
88+
matchRules: >
89+
# TCP access log
90+
# Format: [%START_TIME%] %RESPONSE_FLAGS% %KUMA_MESH% %KUMA_SOURCE_ADDRESS_WITHOUT_PORT%(%KUMA_SOURCE_SERVICE%)->%UPSTREAM_HOST%(%KUMA_DESTINATION_SERVICE%) took %DURATION%ms, sent %BYTES_SENT% bytes, received: %BYTES_RECEIVED% bytes
91+
# Reference: https://github.com/kumahq/kuma/blob/2.11.1/pkg/plugins/policies/meshaccesslog/plugin/xds/configurer.go#L28
92+
# Sample:
93+
# [2025-06-26T21:29:21.535Z] - default 10.42.1.5(unknown)->10.42.1.5:6379(redis_kuma-demo_svc_6379) took 14004ms, sent 6195 bytes, received: 194 bytes
94+
95+
kuma_dp_tcp_log (\[%{isoTimestamp:date}\])? (%{word:response.flags}|-) %{notSpace:kuma.mesh} %{ipOrHost:kuma.source_address_without_port}\((unknown|%{serviceName:kuma.source_service})\)\->(-|%{notOpeningParens:kuma.upstream.host})\((unknown|%{serviceName:kuma.destination_service})\) took %{integer:duration:scale(1000000)}ms, sent %{integer:network.bytes_written} bytes, received: %{integer:network.bytes_read} bytes
96+
97+
# HTTP Access log
98+
# Format: [%START_TIME%] %KUMA_MESH% "%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%" %RESPONSE_CODE% %RESPONSE_FLAGS% %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% "%REQ(X-FORWARDED-FOR)%" "%REQ(USER-AGENT)%" "%REQ(X-B3-TRACEID?X-DATADOG-TRACEID)%" "%REQ(X-REQUEST-ID)%" "%REQ(:AUTHORITY)%" "%KUMA_SOURCE_SERVICE%" "%KUMA_DESTINATION_SERVICE%" "%KUMA_SOURCE_ADDRESS_WITHOUT_PORT%" "%UPSTREAM_HOST%"
99+
# Reference: https://github.com/kumahq/kuma/blob/2.11.1/pkg/plugins/policies/meshaccesslog/plugin/xds/configurer.go#L29
100+
# Trace ID References: https://www.envoyproxy.io/docs/envoy/latest/configuration/http/http_conn_man/headers#x-datadog-trace-id https://github.com/openzipkin/b3-propagation
101+
# Sample:
102+
# [2025-06-26T13:19:18.314Z] default "GET /api?timeout=32s HTTP/1.1" 404 NR 0 0 0 - "10.42.1.26" "kubectl/v1.31.5+k3s1 (linux/arm64) kubernetes/56ec5dd" "-" "0efbfa26-1e2d-9144-919a-e9985d0eab30" "localhost:8080" "unknown" "edge-gateway_kuma-demo_svc" "10.42.3.27" "-"
103+
104+
105+
kuma_dp_http_access_log (\[%{isoTimestamp:date}\])? %{notSpace:kuma.mesh} "%{word:http.method} %{notSpace:http.url} %{httpVersion:http.version}" %{integer:http.status_code} %{word:response.flags} %{integer:network.bytes_read} %{integer:network.bytes_written} %{integer:duration:scale(1000000)} (%{integer:response.x_envoy_upstream_service_time}|-) "%{xForwardedFor:http._x_forwarded_for}" "%{notQuote:http.useragent}" "(-|%{traceId:kuma.trace_id})" "%{uuid:http.request_id}" "%{authority:kuma.http.request.authority}" "(unknown|%{serviceName:kuma.source_service})" "(unknown|%{serviceName:kuma.destination_service})" "(%{ipOrHost:kuma.source_address_without_port}|-)" "(%{ipOrHost:kuma.upstream.host}|-)"
106+
107+
# Log message from a Kuma dataplane
108+
# Samples
109+
# [2025-06-03 18:44:06.650][31][info][upstream] [source/common/listener_manager/lds_api.cc:106] lds: add/update listener 'outbound:10.43.182.159:80'
110+
# [2025-06-03 18:54:23.975][32][info][main] [source/server/server.cc:998] main dispatch loop exited
111+
112+
kuma_dp \[%{date("yyyy-MM-dd HH:mm:ss.SSS"):date}\]\[%{number:logger.thread_name}\]\[%{word:level}\]\[%{word:component}\] \[%{data:logger.file}:%{number:logger.lineno}\] %{data:message}
113+
114+
# Log messages in structured format
115+
# Samples
116+
# 2025-05-26T20:44:41.327Z INFO plugin.runtime.gateway registered gateway plugin
117+
# 2025-05-26T20:47:04.174Z INFO controllers.Service annotating service which is part of the mesh {"service": {"name":"datadog-admission-controller","namespace":"datadog-operator"}, "annotation": "ingress.kubernetes.io/service-upstream=true"}
118+
119+
kuma_cp_structured_log %{isoTimestamp:date}\t%{word:level}\t%{loggerName:logger.name}\t%{notTab:message}(\t%{data::json})?
120+
121+
# Log messages from Kubernetes libraries used by Kuma
122+
# Sample
123+
# I0526 20:44:41.456499 1 leaderelection.go:257] attempting to acquire leader lease kuma-system/cp-leader-lease...
124+
125+
kuma_cp_glog %{regex("\\w"):level}%{date("MMdd HH:mm:ss.SSSSSS"):date}\s+%{number:logger.thread_name} %{loggerName:logger.name}:%{number:logger.lineno}\] %{data:message}
126+
127+
# Log messages from Kuma's built-in HTTP server
128+
# Sample
129+
# 2025/05/31 15:56:10 http: TLS handshake error from 10.42.0.1:41588: EOF
130+
131+
kuma_cp_go_http %{goTimestamp:date}\s%{word:prefix}:\s%{data:message}
132+
133+
# Log message for Kuma dataplane shutdown (SIGTERM)
134+
# Sample:
135+
# [INFO] SIGTERM: Shutting down servers then terminating
136+
137+
kuma_dp_signal \[%{word:level}\] %{word:signal}: %{data:message}
138+
- type: user-agent-parser
139+
name: User-Agent Parser
140+
enabled: true
141+
sources:
142+
- http.useragent
143+
target: http.useragent_details
144+
encoded: false
145+
combineVersionDetails: false
146+
- type: message-remapper
147+
name: Define `message` as the official message of the of the log
148+
enabled: true
149+
sources:
150+
- message
151+
- type: status-remapper
152+
name: Define `level` as the official status of the log
153+
enabled: true
154+
sources:
155+
- level
156+
- type: date-remapper
157+
name: Define 'date' as the official date of the log
158+
enabled: true
159+
sources:
160+
- date

‎kuma/assets/logs/kuma_tests.yaml‎

Lines changed: 157 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
id: kuma
2+
tests:
3+
-
4+
sample: "[2025-06-26T21:29:21.535Z] - default 10.42.1.5(unknown)->10.42.1.5:6379(redis_kuma-demo_svc_6379) took 14004ms, sent 6195 bytes, received: 194 bytes"
5+
result:
6+
custom:
7+
date: "2025-06-26T21:29:21.535Z"
8+
duration: 1.4004E10
9+
kuma:
10+
destination_service: "redis_kuma-demo_svc_6379"
11+
mesh: "default"
12+
source_address_without_port: "10.42.1.5"
13+
upstream:
14+
host: "10.42.1.5:6379"
15+
network:
16+
bytes_read: 194
17+
bytes_written: 6195
18+
message: "[2025-06-26T21:29:21.535Z] - default 10.42.1.5(unknown)->10.42.1.5:6379(redis_kuma-demo_svc_6379) took 14004ms, sent 6195 bytes, received: 194 bytes"
19+
tags:
20+
- "source:LOGS_SOURCE"
21+
timestamp: 1750973361535
22+
-
23+
sample: "[2025-06-26T13:19:18.314Z] default \"GET /api?timeout=32s HTTP/1.1\" 404 NR 0 0 0 - \"10.42.1.26\" \"kubectl/v1.31.5+k3s1 (linux/arm64) kubernetes/56ec5dd\" \"-\" \"0efbfa26-1e2d-9144-919a-e9985d0eab30\" \"localhost:8080\" \"unknown\" \"edge-gateway_kuma-demo_svc\" \"10.42.3.27\" \"-\""
24+
result:
25+
custom:
26+
date: "2025-06-26T13:19:18.314Z"
27+
duration: 0.0
28+
http:
29+
_x_forwarded_for: "10.42.1.26"
30+
method: "GET"
31+
request_id: "0efbfa26-1e2d-9144-919a-e9985d0eab30"
32+
status_code: 404
33+
url: "/api?timeout=32s"
34+
useragent: "kubectl/v1.31.5+k3s1 (linux/arm64) kubernetes/56ec5dd"
35+
useragent_details:
36+
browser:
37+
family: "Other"
38+
device:
39+
category: "Desktop"
40+
family: "Other"
41+
os:
42+
family: "Linux"
43+
version: "HTTP/1.1"
44+
kuma:
45+
destination_service: "edge-gateway_kuma-demo_svc"
46+
http:
47+
request:
48+
authority: "localhost:8080"
49+
mesh: "default"
50+
source_address_without_port: "10.42.3.27"
51+
network:
52+
bytes_read: 0
53+
bytes_written: 0
54+
response:
55+
flags: "NR"
56+
message: "[2025-06-26T13:19:18.314Z] default \"GET /api?timeout=32s HTTP/1.1\" 404 NR 0 0 0 - \"10.42.1.26\" \"kubectl/v1.31.5+k3s1 (linux/arm64) kubernetes/56ec5dd\" \"-\" \"0efbfa26-1e2d-9144-919a-e9985d0eab30\" \"localhost:8080\" \"unknown\" \"edge-gateway_kuma-demo_svc\" \"10.42.3.27\" \"-\""
57+
tags:
58+
- "source:LOGS_SOURCE"
59+
timestamp: 1750943958314
60+
-
61+
sample: "[2025-06-03 18:44:06.650][31][info][upstream] [source/common/listener_manager/lds_api.cc:106] lds: add/update listener 'outbound:10.43.182.159:80'"
62+
result:
63+
custom:
64+
component: "upstream"
65+
date: 1748976246650
66+
level: "info"
67+
logger:
68+
file: "source/common/listener_manager/lds_api.cc"
69+
lineno: 106.0
70+
thread_name: 31.0
71+
message: "lds: add/update listener 'outbound:10.43.182.159:80'"
72+
status: "info"
73+
tags:
74+
- "source:LOGS_SOURCE"
75+
timestamp: 1748976246650
76+
-
77+
sample: "[2025-06-03 18:54:23.975][32][info][main] [source/server/server.cc:998] main dispatch loop exited"
78+
result:
79+
custom:
80+
component: "main"
81+
date: 1748976863975
82+
level: "info"
83+
logger:
84+
file: "source/server/server.cc"
85+
lineno: 998.0
86+
thread_name: 32.0
87+
message: "main dispatch loop exited"
88+
status: "info"
89+
tags:
90+
- "source:LOGS_SOURCE"
91+
timestamp: 1748976863975
92+
-
93+
sample: "2025-05-26T20:44:41.327Z\tINFO\tplugin.runtime.gateway\tregistered gateway plugin"
94+
result:
95+
custom:
96+
date: "2025-05-26T20:44:41.327Z"
97+
level: "INFO"
98+
logger:
99+
name: "plugin.runtime.gateway"
100+
message: "registered gateway plugin"
101+
status: "info"
102+
tags:
103+
- "source:LOGS_SOURCE"
104+
timestamp: 1748292281327
105+
-
106+
sample: "2025-05-26T20:47:04.174Z\tINFO\tcontrollers.Service\tannotating service which is part of the mesh\t{\"service\": {\"name\":\"datadog-admission-controller\",\"namespace\":\"datadog-operator\"}, \"annotation\": \"ingress.kubernetes.io/service-upstream=true\"}"
107+
result:
108+
custom:
109+
annotation: "ingress.kubernetes.io/service-upstream=true"
110+
date: "2025-05-26T20:47:04.174Z"
111+
level: "INFO"
112+
logger:
113+
name: "controllers.Service"
114+
service:
115+
name: "datadog-admission-controller"
116+
namespace: "datadog-operator"
117+
message: "annotating service which is part of the mesh"
118+
status: "info"
119+
tags:
120+
- "source:LOGS_SOURCE"
121+
timestamp: 1748292424174
122+
-
123+
sample: "I0526 20:44:41.456499 1 leaderelection.go:257] attempting to acquire leader lease kuma-system/cp-leader-lease..."
124+
result:
125+
custom:
126+
date: 44138681456
127+
level: "I"
128+
logger:
129+
lineno: 257.0
130+
name: "leaderelection.go"
131+
thread_name: 1.0
132+
message: "attempting to acquire leader lease kuma-system/cp-leader-lease..."
133+
status: "info"
134+
tags:
135+
- "source:LOGS_SOURCE"
136+
timestamp: 44138681456
137+
-
138+
sample: "2025/05/31 15:56:10 http: TLS handshake error from 10.42.0.1:41588: EOF"
139+
result:
140+
custom:
141+
date: "2025/05/31 15:56:10"
142+
prefix: "http"
143+
message: "TLS handshake error from 10.42.0.1:41588: EOF"
144+
tags:
145+
- "source:LOGS_SOURCE"
146+
timestamp: 1
147+
-
148+
sample: "[INFO] SIGTERM: Shutting down servers then terminating"
149+
result:
150+
custom:
151+
level: "INFO"
152+
signal: "SIGTERM"
153+
message: "Shutting down servers then terminating"
154+
status: "info"
155+
tags:
156+
- "source:LOGS_SOURCE"
157+
timestamp: 1

‎kuma/assets/saved_views/logs_overview.json‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
{
2+
"name": "Kuma Mesh Access Logs Overview",
3+
"type": "logs",
4+
"page": "stream",
5+
"query": "source:kuma @kuma.destination_service:*",
6+
"timerange": {
7+
"interval_ms": 3600000
8+
},
9+
"visible_facets": [
10+
"source",
11+
"host",
12+
"service",
13+
"@kuma.destination_service",
14+
"@kuma.source_service",
15+
"@kuma.source_address_without_port",
16+
"@kuma.mesh",
17+
"@kuma.upstream.host"
18+
],
19+
"options": {
20+
"show_date_column": true,
21+
"show_message_column": true,
22+
"message_display": "inline",
23+
"show_timeline": true
24+
}
25+
}

‎kuma/manifest.json‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,12 @@
4747
"kuma-dp"
4848
]
4949
},
50+
"saved_views": {
51+
"Kuma Mesh Access Logs Overview": "assets/saved_views/logs_overview.json"
52+
},
53+
"logs": {
54+
"source": "kuma"
55+
},
5056
"dashboards": {
5157
"Kuma Control Plane": "assets/dashboards/kuma_control_plane.json",
5258
"Kuma Service Communication": "assets/dashboards/kuma_service_communication.json"

0 commit comments

Comments
 (0)