Skip to content

Distributed table materialization fails with replicated database #673

Description

@raffael-gr

distributed_table and distributed_incremental materializations false-positive raise "cluster setting must be set" when used against a database created with ENGINE = Replicated(...), even when cluster: is correctly configured in the profile.

After #447, on_cluster_clause(...) correctly returns an empty string for Replicated databases — the Replicated database engine broadcasts DDL via ZooKeeper, and ClickHouse rejects ON CLUSTER against it (Code: 80. ON CLUSTER is not allowed for Replicated
database). All other DDL paths in the package (CREATE / DROP / INSERT / ALTER / grants / EXCHANGE / persist_docs / etc.) handle this correctly because they only consult on_cluster_clause's output.

Two leftover guards weren't updated when on_cluster_clause semantics changed:

  • dbt/include/clickhouse/macros/materializations/distributed_table.sql:13-16
  • dbt/include/clickhouse/macros/materializations/incremental/distributed_incremental.sql:13-16
{% set on_cluster = on_cluster_clause(target_relation) %}
{% if on_cluster.strip() == '' %}
   {% do exceptions.raise_compiler_error('To use distributed materialization cluster setting in dbt profile must be set') %}
{% endif %}

These check whether on_cluster_clause returned text, not whether cluster is configured. Post-#447 those are no longer the same condition: cluster can be set and on_cluster_clause legitimately empty (Replicated DB). The guards should consult
adapter.get_clickhouse_cluster_name() instead — that returns None only when cluster: is genuinely unset.

Steps to reproduce

  1. Connect to a ClickHouse instance and create a Replicated database
  2. Configure profiles.yml with cluster: and database_engine: Replicated.
  3. Add any model with {{ config(materialized='distributed_table') }} and run dbt run.

Expected behaviour

Model materializes successfully — _local ReplicatedMergeTree per shard plus a Distributed wrapper, no ON CLUSTER clause anywhere. The Replicated DB engine handles DDL broadcast.

Actual: Compilation Error: To use distributed materialization cluster setting in dbt profile must be set, even though cluster is set.

Code examples, such as models or profile settings

  profiles.yml:
  my_project:
    target: dev
    outputs:
      dev:
        type: clickhouse
        host: clickhouse.example.com
        port: 443
        user: my_user
        password: "{{ env_var('CH_PASSWORD') }}"
        schema: my_repl_db
        secure: true
        driver: http
        cluster: my_cluster
        database_engine: Replicated
        custom_settings:
          insert_distributed_sync: 1

Minimal model models/sharded.sql:

  {{ config(materialized='distributed_table', engine='ReplicatedMergeTree()', sharding_key='rand()', order_by='id') }}
  select 1 as id

Suggested fix — replace the guard in both files with:

  {% if adapter.get_clickhouse_cluster_name() is none %}
     {% do exceptions.raise_compiler_error('To use distributed materialization cluster setting in dbt profile must be set') %}
  {% endif %}

or simply delete it: create_distributed_table (called below) already raises 'Cluster name should be defined for using distributed materializations' when cluster is genuinely missing, so the early guard is redundant.

dbt and/or ClickHouse server logs

  Compilation Error in model sharded (models/sharded.sql)
    To use distributed materialization cluster setting in dbt profile must be set

    > in macro materialization_distributed_table_clickhouse (macros/materializations/distributed_table.sql)
    > called by model sharded (models/sharded.sql)

Environment

  • dbt version: 1.11.11
  • dbt-clickhouse version: 1.10.0
  • clickhouse-connect version (if using http): 1.1.1
  • Python version: 3.13.12
  • Operating system: Linux

ClickHouse server

  • ClickHouse Server version: 25.10.7.6
  • ClickHouse Server non-default settings: target database created with ENGINE = Replicated(...); standard <remote_servers> cluster configuration.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions