Add random suffix to incremental new data table (#152)

* Add random suffix to incremental new data table

* Update test matrix

* Fix typo

* Fix test release years

Author: genzgd

.github/workflows/test_matrix.yml

@@ -28,10 +28,10 @@ jobs:
           - '3.10'
           - '3.11'
         clickhouse-version:
-          - '22.3'
           - '22.8'
-          - '22.11'
-          - '22.12'
+          - '23.2'
+          - '23.3'
+          - '23.4'
           - latest
 
     steps:

CHANGELOG.md

@@ -1,3 +1,11 @@
+### Release [1.4.1], 2023-05-11
+#### Improvements
+- Reduce the number of SQL calls for Modify Comment operations.  Thanks to [Konstantin Ilchenko](https://github.com/simpl1g).
+- Add "on cluster" to several additional macros to better support distributed tables.  Thanks to [Saurabh Bikram](https://github.com/saurabhbikram)
+- Add the unique invocation id to temporary "new data" used in `delete+insert` incremental materializations to allow parallel transformations on the same
+table.  In general parallel transformations are risky, so this approach should only be used when transformations are explicitly limited
+to non-overlapping data ranges.  Closes https://github.com/ClickHouse/dbt-clickhouse/issues/150
+
 ### Release [1.4.0], 2023-02-06
 #### Improvements
 - Support dbt [1.4.1] https://github.com/ClickHouse/dbt-clickhouse/issues/135

dbt/include/clickhouse/macros/materializations/incremental.sql

@@ -178,7 +178,8 @@
 
 
 {% macro clickhouse__incremental_delete_insert(existing_relation, unique_key, incremental_predicates) %}
-    {% set new_data_relation = existing_relation.incorporate(path={"identifier": model['name'] + '__dbt_new_data'}) %}
+    {% set new_data_relation = existing_relation.incorporate(path={"identifier": model['name']
+       + '__dbt_new_data_' + invocation_id.replace('-', '_')}) %}
     {{ drop_relation_if_exists(new_data_relation) }}
     {% call statement('main') %}
         {{ get_create_table_as_sql(False, new_data_relation, sql) }}

dev_requirements.txt

@@ -1,5 +1,5 @@
 dbt-core~=1.4.1
-clickhouse-connect>=0.5.5
+clickhouse-connect>=0.5.24
 clickhouse-driver>=0.2.3
 pytest>=7.2.0
 pytest-dotenv==0.5.2

setup.py

@@ -55,7 +55,7 @@ def _dbt_clickhouse_version():
     },
     install_requires=[
         f'dbt-core~={dbt_version}',
-        'clickhouse-connect>=0.5.5',
+        'clickhouse-connect>=0.5.24',
         'clickhouse-driver>=0.2.3',
     ],
     python_requires=">=3.7",