Skip to content

Commit 2a5ad5e

Browse files
sangeetashivajisangeetashivaji
authored
[DBMON-6018] ClickHouse support for DBM (DataDog#22341)
* Add dbm integration for clickhouse * Update * Update to the correct implementation * Update implementation * Update * Update * Update * Update * Add new metadata payloads * Fix avg & count metrics * Update clickhouse implementation * Update changelog * Remove cloud metadata for clickhouse * Cleanup cloud metadata * Update metrics logic * Adding query completion payloads * Revert "Update metrics logic" This reverts commit 857db34. * Fixes * Update metrics logic for checkpointing logic * clusterAllReplica changes * Bug fixes * Update query with internal user exclusion * Fix: CI build issues * Fix: Skip DBM tests on unsupported ClickHouse versions (<21.8) * Update changelog * Fix: Update test assertions for query_log_table placeholder * Fix tests * Fixing CI * Fixing CI * Fixing CI * Revert accidental change to datadog_checks_dev * Fix SharedConfig model: add default values for optional fields * Fix config models: add default values for Optional fields * Trigger CI * Fix shared.py to match ddev generated format with = None defaults * Rename clickhouse_cloud to single_endpoint_mode for more generic naming * Update conf.yaml.example to use single_endpoint_mode * Fix config model ordering and checkpoint error handling * Fix trailing whitespace in conf.yaml.example * Fix ruff line length formatting * Use tag manager to regularize tag management * Fixing errors * Cleanup code to have only query activity * Updating change log * Address comments * Update * Update * Update * Update * Address comments - 2 * Address comments * Address comments * Address comments * Address comments * chore: retrigger CI * Address comments * Advance checkpoint regardless of success * Update activity to samples accross clickhouse integration * Fix CI * Undo changes * Health events using Feature class * Update Health events using Feature class * Update config and instance * Rename query completes * Updates to query complete rename * Update * Removing peak_memory_usage field
1 parent f88db5b commit 2a5ad5e

23 files changed

Lines changed: 4171 additions & 56 deletions

‎clickhouse/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -115,4 +115,4 @@ Need help? Contact [Datadog support][10].
115115
[7]: https://docs.datadoghq.com/agent/guide/agent-commands/#agent-status-and-information
116116
[8]: https://github.com/DataDog/integrations-core/blob/master/clickhouse/metadata.csv
117117
[9]: https://github.com/DataDog/integrations-core/blob/master/clickhouse/assets/service_checks.json
118-
[10]: https://docs.datadoghq.com/help/
118+
[10]: https://docs.datadoghq.com/help/

‎clickhouse/assets/configuration/spec.yaml‎

Lines changed: 154 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -21,11 +21,14 @@ files:
2121
value:
2222
type: string
2323
- name: port
24-
description: The port used to connect to the system.
24+
description: |
25+
The port used to connect to the system.
26+
This integration uses the HTTP interface, so the default is 8123 (HTTP port).
27+
Note: Port 9000 is for ClickHouse's native TCP protocol, which is not used by this integration.
2528
fleet_configurable: true
2629
value:
2730
type: integer
28-
example: 9000
31+
example: 8123
2932
- name: username
3033
description: The database user to authenticate as.
3134
fleet_configurable: true
@@ -60,7 +63,7 @@ files:
6063
description: |
6164
The compression algorithm to use. The default is no compression.
6265
If br is specified, the brotli library must be installed separately.
63-
66+
6467
Valid values are:
6568
- lz4
6669
- zstd
@@ -83,6 +86,154 @@ files:
8386
value:
8487
type: boolean
8588
example: True
89+
- name: dbm
90+
description: |
91+
Enable Database Monitoring (DBM) to collect query samples, metrics, and completions.
92+
This feature provides deep observability into query performance.
93+
value:
94+
type: boolean
95+
example: false
96+
- name: database_identifier
97+
description: |
98+
Controls how the database is identified. Each unique value of this identifier will appear in DBM as a separate
99+
billable instance.
100+
101+
The default value is "$server:$port:$db" which uniquely identifies a ClickHouse database by its server, port,
102+
and database name.
103+
104+
This value will be used as-is for the display name of the instance but will be normalized
105+
when applied as the `database_instance` tag. Please see https://docs.datadoghq.com/getting_started/tagging/
106+
for more details on Datadog tag normalization.
107+
options:
108+
- name: template
109+
description: |
110+
The template string for the database identifier. Supports variable substitution using $variable syntax.
111+
Available variables: $server, $port, $db, plus any custom tags (e.g., $env, $region).
112+
value:
113+
type: string
114+
display_default: "$server:$port:$db"
115+
example: "$env-$server:$port:$db"
116+
- name: single_endpoint_mode
117+
description: Set to `true` when connecting through a single endpoint for ClickHouse Cloud.
118+
value:
119+
type: boolean
120+
example: false
121+
- name: query_samples
122+
description: Configure collection of database samples snapshots from system.processes
123+
options:
124+
- name: enabled
125+
description: |
126+
Enable collection of database samples snapshots. Requires `dbm: true`.
127+
Samples snapshots show currently executing queries and connection counts.
128+
value:
129+
type: boolean
130+
example: true
131+
- name: collection_interval
132+
description: |
133+
Set the samples snapshot collection interval (in seconds). Each collection involves a single query to
134+
`system.processes` to capture currently executing queries.
135+
value:
136+
type: number
137+
example: 1
138+
- name: payload_row_limit
139+
description: |
140+
Set the maximum number of active sessions to include in each samples snapshot.
141+
value:
142+
type: integer
143+
example: 1000
144+
- name: run_sync
145+
hidden: true
146+
description: |
147+
Run the samples collection synchronously. This is useful for testing purposes, but should not be used
148+
in production as it can block the main thread and cause performance issues.
149+
value:
150+
type: boolean
151+
example: false
152+
- name: query_metrics
153+
description: Configure collection of query metrics
154+
options:
155+
- name: enabled
156+
description: |
157+
Enable collection of query metrics. Requires `dbm: true`.
158+
value:
159+
type: boolean
160+
example: true
161+
- name: collection_interval
162+
description: |
163+
Set the query metric collection interval (in seconds). Each collection involves a single query to
164+
`system.query_log`. If a non-default value is chosen then that exact same value must be used for *every*
165+
check instance. Running different instances with different collection intervals is not supported.
166+
value:
167+
type: number
168+
example: 10
169+
- name: run_sync
170+
hidden: true
171+
description: |
172+
Run the query metrics collection synchronously. This is useful for testing purposes, but should not be used
173+
in production as it can block the main thread and cause performance issues.
174+
value:
175+
type: boolean
176+
example: false
177+
- name: full_statement_text_cache_max_size
178+
hidden: true
179+
description: |
180+
Set the max size of the cache used for the full statement text.
181+
value:
182+
type: number
183+
default: 10000
184+
- name: full_statement_text_samples_per_hour_per_query
185+
hidden: true
186+
description: |
187+
Set the max number of full statement text samples to collect per hour per query.
188+
value:
189+
type: number
190+
default: 1
191+
- name: query_completions
192+
description: Configure collection of query completions from system.query_log
193+
options:
194+
- name: enabled
195+
description: |
196+
Enable collection of query completions. Requires `dbm: true`.
197+
These samples represent individual completed queries and appear on the DBM Query Samples page.
198+
value:
199+
type: boolean
200+
example: true
201+
- name: collection_interval
202+
description: |
203+
Set the query completions collection interval (in seconds).
204+
Lower values show more recent queries but increase collection overhead.
205+
value:
206+
type: number
207+
example: 10
208+
- name: samples_per_hour_per_query
209+
description: |
210+
Set the maximum number of samples to collect per hour per unique query signature.
211+
This rate limiting prevents excessive data collection for frequently executed queries.
212+
value:
213+
type: number
214+
example: 15
215+
- name: seen_samples_cache_maxsize
216+
hidden: true
217+
description: |
218+
Set the max size of the cache used for rate limiting samples.
219+
value:
220+
type: number
221+
default: 10000
222+
- name: max_samples_per_collection
223+
hidden: true
224+
description: |
225+
Maximum number of samples to collect in a single run (applies LIMIT in SQL).
226+
Prevents overwhelming the agent with too many samples at once.
227+
value:
228+
type: number
229+
default: 1000
230+
- name: run_sync
231+
hidden: true
232+
description: |
233+
Run the query completions collection synchronously. For testing only.
234+
value:
235+
type: boolean
236+
example: false
86237
- template: instances/db
87238
overrides:
88239
custom_queries.value.example:

‎clickhouse/changelog.d/22341.added‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Add ClickHouse support for DBM with Query Metrics, Query Samples & Query Completions

0 commit comments

Comments
 (0)