Add a hosting_type tag distinguishing ClickHouse Cloud from self-hosted (DataDog#24736)

* Add a hosting_type tag distinguishing ClickHouse Cloud from self-hosted

DBM had no way to tell a ClickHouse Cloud service apart from a self-hosted
deployment. The only Cloud signal was single_endpoint_mode, a flag the operator
sets by hand, so slicing metrics, samples, or instance metadata by hosting model
was not possible even though the two behave differently: SharedMergeTree vs
ReplicatedMergeTree, clusterAllReplicas topology, per-node checkpointing, and
datetime vs integer types in system.query_log.

Resolves the hosting model from two independent server-side signals and attaches
it as an instance-level tag, following the same shape as clickhouse_cluster:
resolved once per check instance, set on the tag manager before the query manager
is built, so it flows through the existing payload tags/ddtags plumbing to query
metrics, samples, completions, errors, parts and merges, FQT and instance
metadata. The tag key and value vocabulary follow the mongo integration, the
other DBM check that reports a hosting model.

Both signals must agree before reporting cloud. cloud_mode is read from
system.settings rather than via getSetting('cloud_mode'), which raises on the
versions predating the setting and would increment FailedQuery/FailedSelectQuery.
SharedMergeTree is looked up in system.table_engines rather than system.tables so
a freshly provisioned Cloud service with no user tables still resolves, and both
filter with exact equality rather than LIKE, which makes ClickHouse compile a re2
regex and bump RegexpCreated. Both lessons come from the clickhouse_cluster work.

A probe that succeeds without finding its marker is a definite negative and
settles the conjunction on its own. A probe that raises is indeterminate and
yields 'unknown' rather than a self-hosted verdict built on nothing more than a
permission error or a system table the server is too old to have.

Verified against a live ClickHouse Cloud service, which resolves to
clickhouse-cloud, and against self-hosted 18 and 24.8, which resolve to
self-hosted through the setting-absent and setting-disabled paths respectively.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Add changelog entry

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Apply ruff format to the hosting type test

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Describe the hosting type probes as unconditional

The docstring said a negative signal 'short-circuits to self-hosted', which reads as
control flow that does not exist: both probes are always issued and the conjunction is
evaluated afterwards. Say so, and record why neither probe is skipped.

Also correct the stated failure mode. A lack of privileges was listed as a reason a probe
might raise, but neither system.settings nor system.table_engines is gated behind grants:
a user granted only SELECT on one database still reads all 1091 settings rows and all 64
table engines, verified against a live 24.8 cluster. The indeterminate case is a transport
error or a server too old to have the table, not a permissions gap.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* Trim verbose comments and docstrings on the hosting_type change

* Remove two more redundant comments on the hosting_type change

* Drop redundant _hosting_type_resolved flag

hosting_type always resolves to a non-empty string, so None alone
distinguishes "not yet resolved" without a second boolean.

* Move hosting_type tag into the expected_tags literal

It's unconditional, unlike the cluster tag, so it belongs with the
other always-present tags rather than as a trailing append.

---------

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>

Author: sangeetashivaji

clickhouse/changelog.d/24736.added

@@ -0,0 +1 @@
+Add a `hosting_type` tag identifying whether the ClickHouse instance is ClickHouse Cloud or self-hosted.

clickhouse/datadog_checks/clickhouse/clickhouse.py

@@ -24,10 +24,14 @@
 from .statements import ClickhouseStatementMetrics
 from .table_metrics import ClickhouseTableMetrics
 from .utils import (
+    CLOUD_MODE_QUERY,
     CLUSTER_MACRO_QUERY,
     CLUSTER_NAME_QUERY,
     CLUSTER_TAG,
+    HOSTING_TYPE_TAG,
+    SHARED_MERGE_TREE_QUERY,
     ErrorSanitizer,
+    HostingType,
     cluster_aware_query,
 )
 
@@ -68,6 +72,7 @@ def __init__(self, name, init_config, instances):
         self._dbms_version = None
         self._cluster_name = None
         self._cluster_name_resolved = False
+        self._hosting_type = None
 
         # Track last emission time for database instance metadata (rate limiting)
         self._database_instance_last_emitted = 0
@@ -255,6 +260,7 @@ def check(self, _):
         # self.tags below, since both snapshot the tag list.
         if self.cluster_name:
             self.tag_manager.set_tag(CLUSTER_TAG, self.cluster_name, replace=True)
+        self.tag_manager.set_tag(HOSTING_TYPE_TAG, self.hosting_type, replace=True)
 
         if self._query_manager is None or self._query_manager_version != self._server_version:
             self._query_manager = self._build_query_manager()
@@ -397,6 +403,45 @@ def _resolve_cluster_name(self) -> str | None:
         self.log.debug('No ClickHouse cluster name found; %s tag will not be emitted', CLUSTER_TAG)
         return None
 
+    @property
+    def hosting_type(self) -> str:
+        """Whether this instance is ClickHouse Cloud or self-hosted, cached after the first check run."""
+        if self._hosting_type is None:
+            self._hosting_type = self._resolve_hosting_type()
+        return self._hosting_type
+
+    def _resolve_hosting_type(self) -> str:
+        """Combine two independent Cloud signals; both must agree to report cloud, either can veto it."""
+        cloud_mode = self._probe_cloud_mode()
+        shared_merge_tree = self._probe_shared_merge_tree()
+        self.log.debug('Hosting type signals: cloud_mode=%s, shared_merge_tree=%s', cloud_mode, shared_merge_tree)
+
+        if cloud_mode is False or shared_merge_tree is False:
+            return HostingType.SELF_HOSTED
+        if cloud_mode and shared_merge_tree:
+            return HostingType.CLOUD
+        return HostingType.UNKNOWN
+
+    def _probe_cloud_mode(self) -> bool | None:
+        """Whether the server reports cloud_mode enabled, or None when the probe failed."""
+        try:
+            rows = self.execute_query_raw(CLOUD_MODE_QUERY)
+            if not rows or not rows[0]:
+                return False
+            return str(rows[0][0]) not in ('', '0')
+        except Exception as e:
+            self.log.debug('Unable to read the cloud_mode setting: %s', e)
+            return None
+
+    def _probe_shared_merge_tree(self) -> bool | None:
+        """Whether the Cloud-only SharedMergeTree engine exists, or None when the probe failed."""
+        try:
+            rows = self.execute_query_raw(SHARED_MERGE_TREE_QUERY)
+            return bool(rows and rows[0] and int(rows[0][0]) > 0)
+        except Exception as e:
+            self.log.debug('Unable to check for the SharedMergeTree engine: %s', e)
+            return None
+
     @property
     def database_identifier_template(self) -> str:
         return self._config.database_identifier.template

clickhouse/datadog_checks/clickhouse/utils.py

@@ -74,6 +74,22 @@ def compact_query(query):
 )
 
 
+HOSTING_TYPE_TAG = 'hosting_type'
+
+
+class HostingType:
+    CLOUD = 'clickhouse-cloud'
+    SELF_HOSTED = 'self-hosted'
+    UNKNOWN = 'unknown'
+
+
+# system.settings avoids raising on versions predating cloud_mode (before 23.x).
+CLOUD_MODE_QUERY = "SELECT value FROM system.settings WHERE name = 'cloud_mode'"
+
+# table_engines lists supported engines even before any tables exist; exact match avoids a LIKE regex compile.
+SHARED_MERGE_TREE_QUERY = "SELECT count() FROM system.table_engines WHERE name = 'SharedMergeTree'"
+
+
 def cluster_aware_query(base: dict) -> dict:
     """Build a cluster-aware variant that reads all replicas and tags each row per node.
 

clickhouse/tests/test_clickhouse.py

@@ -4,7 +4,7 @@
 import pytest
 
 from datadog_checks.clickhouse import ClickhouseCheck
-from datadog_checks.clickhouse.utils import CLUSTER_TAG
+from datadog_checks.clickhouse.utils import CLUSTER_TAG, HOSTING_TYPE_TAG
 from datadog_checks.dev.utils import get_metadata_metrics
 
 from . import common
@@ -55,6 +55,7 @@ def test_custom_queries(aggregator, instance, dd_run_check):
         'test:clickhouse',
         'database_hostname:{}'.format(check.database_hostname),
         'database_instance:{}:{}:default'.format(instance['server'], instance['port']),
+        '{}:{}'.format(HOSTING_TYPE_TAG, check.hosting_type),
     ]
     # ClickHouse ships a built-in is_local `default` cluster on some versions (and Cloud reports one
     # too), so every metric carries the clickhouse_cluster tag when a cluster resolves.

clickhouse/tests/test_unit.py

@@ -9,10 +9,14 @@
 from datadog_checks.clickhouse import ClickhouseCheck, advanced_queries, queries
 from datadog_checks.clickhouse.utils import (
     BUILTIN_SAMPLE_CLUSTERS,
+    CLOUD_MODE_QUERY,
     CLUSTER_GROUP_PREFIX,
     CLUSTER_MACRO_QUERY,
     CLUSTER_NAME_QUERY,
     CLUSTER_TAG,
+    HOSTING_TYPE_TAG,
+    SHARED_MERGE_TREE_QUERY,
+    HostingType,
     cluster_aware_query,
 )
 
@@ -464,8 +468,11 @@ def test_get_queries_uses_base_queries_for_direct_connection(instance, use_advan
     assert all('clusterAllReplicas' not in q['query'] for q in check.get_queries())
 
 
-def make_cluster_name_check(query_results):
-    """Build a check whose execute_query_raw replays query_results keyed by SQL."""
+def make_query_replaying_check(query_results):
+    """Build a check whose execute_query_raw replays query_results keyed by SQL.
+
+    An Exception value is raised instead of returned, to simulate a failed probe.
+    """
     check = ClickhouseCheck('clickhouse', {}, [BASE_INSTANCE])
 
     def execute(query):
@@ -479,7 +486,7 @@ def execute(query):
 
 
 def test_cluster_name_prefers_the_macro():
-    check = make_cluster_name_check({CLUSTER_MACRO_QUERY: [['macro_cluster']]})
+    check = make_query_replaying_check({CLUSTER_MACRO_QUERY: [['macro_cluster']]})
 
     assert check.cluster_name == 'macro_cluster'
     # system.clusters must not be consulted once the macro answers.
@@ -495,7 +502,7 @@ def test_cluster_name_prefers_the_macro():
     ],
 )
 def test_cluster_name_falls_back_to_system_clusters(macro_result):
-    check = make_cluster_name_check({CLUSTER_MACRO_QUERY: macro_result, CLUSTER_NAME_QUERY: [['prod_cluster']]})
+    check = make_query_replaying_check({CLUSTER_MACRO_QUERY: macro_result, CLUSTER_NAME_QUERY: [['prod_cluster']]})
 
     assert check.cluster_name == 'prod_cluster'
 
@@ -524,7 +531,7 @@ def test_cluster_name_query_excludes_cloud_group_pseudo_cluster():
 
 
 def test_cluster_name_absent_when_both_sources_fail():
-    check = make_cluster_name_check(
+    check = make_query_replaying_check(
         {
             CLUSTER_MACRO_QUERY: Error('query failed'),
             CLUSTER_NAME_QUERY: [],
@@ -536,7 +543,7 @@ def test_cluster_name_absent_when_both_sources_fail():
 
 
 def test_cluster_name_is_cached_including_the_absent_case():
-    check = make_cluster_name_check({CLUSTER_MACRO_QUERY: [], CLUSTER_NAME_QUERY: []})
+    check = make_query_replaying_check({CLUSTER_MACRO_QUERY: [], CLUSTER_NAME_QUERY: []})
 
     assert check.cluster_name is None
     assert check.cluster_name is None
@@ -582,3 +589,64 @@ def test_check_omits_cluster_tag_when_unresolved(instance):
             check.check({})
 
     assert not any(tag.startswith(f'{CLUSTER_TAG}:') for tag in check.tags)
+
+
+PROBE_FAILED = Error('Not enough privileges')
+
+
+@pytest.mark.parametrize(
+    'cloud_mode, shared_merge_tree, expected',
+    [
+        pytest.param([['1']], [[1]], HostingType.CLOUD, id='cloud'),
+        # A negative signal decides on its own, even if the other never answered.
+        pytest.param([], [[1]], HostingType.SELF_HOSTED, id='self-hosted-setting-absent'),
+        pytest.param([['0']], [[1]], HostingType.SELF_HOSTED, id='self-hosted-cloud-mode-off'),
+        pytest.param([['']], [[1]], HostingType.SELF_HOSTED, id='self-hosted-cloud-mode-empty'),
+        pytest.param([['1']], [[0]], HostingType.SELF_HOSTED, id='self-hosted-no-shared-merge-tree'),
+        pytest.param(PROBE_FAILED, [[0]], HostingType.SELF_HOSTED, id='self-hosted-despite-failed-probe'),
+        pytest.param([['0']], PROBE_FAILED, HostingType.SELF_HOSTED, id='self-hosted-despite-failed-engine-probe'),
+        # A failed probe is indeterminate, not a negative.
+        pytest.param(PROBE_FAILED, [[1]], HostingType.UNKNOWN, id='unknown-cloud-mode-unreadable'),
+        pytest.param([['1']], PROBE_FAILED, HostingType.UNKNOWN, id='unknown-engines-unreadable'),
+        pytest.param(PROBE_FAILED, PROBE_FAILED, HostingType.UNKNOWN, id='unknown-both-unreadable'),
+    ],
+)
+def test_hosting_type_resolution(cloud_mode, shared_merge_tree, expected):
+    check = make_query_replaying_check({CLOUD_MODE_QUERY: cloud_mode, SHARED_MERGE_TREE_QUERY: shared_merge_tree})
+
+    assert check.hosting_type == expected
+
+
+def test_hosting_type_is_cached_including_the_unknown_case():
+    check = make_query_replaying_check({CLOUD_MODE_QUERY: PROBE_FAILED, SHARED_MERGE_TREE_QUERY: PROBE_FAILED})
+
+    assert check.hosting_type == HostingType.UNKNOWN
+    assert check.hosting_type == HostingType.UNKNOWN
+
+    # One attempt per signal, not per access: a server that cannot answer is not re-asked.
+    assert check.execute_query_raw.call_count == 2
+
+
+def test_check_tags_with_hosting_type(instance):
+    check = ClickhouseCheck('clickhouse', {}, [instance])
+    with mock.patch.object(ClickhouseCheck, 'cluster_name', new_callable=mock.PropertyMock) as cluster_name:
+        cluster_name.return_value = None
+        with mock.patch.object(ClickhouseCheck, 'hosting_type', new_callable=mock.PropertyMock) as hosting_type:
+            hosting_type.return_value = HostingType.CLOUD
+            with mock.patch('clickhouse_connect.get_client'):
+                check.check({})
+
+    assert f'{HOSTING_TYPE_TAG}:{HostingType.CLOUD}' in check.tags
+
+
+def test_check_always_emits_a_hosting_type_tag(instance):
+    """Unlike the cluster tag, this one has a value for every outcome, so it is never omitted."""
+    check = ClickhouseCheck('clickhouse', {}, [instance])
+    with mock.patch.object(ClickhouseCheck, 'cluster_name', new_callable=mock.PropertyMock) as cluster_name:
+        cluster_name.return_value = None
+        with mock.patch.object(ClickhouseCheck, 'hosting_type', new_callable=mock.PropertyMock) as hosting_type:
+            hosting_type.return_value = HostingType.UNKNOWN
+            with mock.patch('clickhouse_connect.get_client'):
+                check.check({})
+
+    assert f'{HOSTING_TYPE_TAG}:{HostingType.UNKNOWN}' in check.tags