Skip to content

feat: support TTL as a column-level configuration #442

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,32 @@
### Unreleased

#### Improvements

* Added support for [TTL (time-to-live)](https://clickhouse.com/docs/guides/developer/ttl) as a column configuration for `table` and `ephemeral` materializations. This feature is implemented as a [custom constraint](https://docs.getdbt.com/reference/resource-properties/constraints#custom-constraints), which requires model contracts to be enforced. For example:

```sql
-- test_ttl.sql
{{ config(order_by='(ts)', engine='MergeTree()', materialized='table') }}

SELECT now() AS ts,
'Some value that should expire!' AS col_ttl
```

```yaml
models:
- name: test_ttl
description: 'Testing column-level TTL'
config:
contract:
enforced: true
columns:
- name: ts
data_type: timestamp
- name: col_ttl
data_type: String
ttl: ts + INTERVAL 1 DAY
```

### Release [1.8.9], 2025-02-16

#### Improvements
Expand Down
23 changes: 22 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -123,9 +123,30 @@ your_profile_name:

## Column Configuration

> **_NOTE:_** The column configuration options below require [model contracts](https://docs.getdbt.com/docs/collaborate/govern/model-contracts) to be enforced.

| Option | Description | Default if any |
|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------|
| codec | A string consisting of arguments passed to `CODEC()` in the column's DDL. For example: `codec: "Delta, ZSTD"` will be interpreted as `CODEC(Delta, ZSTD)`. | |
| codec | A string consisting of arguments passed to `CODEC()` in the column's DDL. For example: `codec: "Delta, ZSTD"` will be compiled as `CODEC(Delta, ZSTD)`. |
| ttl | A string consisting of a [TTL (time-to-live) expression](https://clickhouse.com/docs/guides/developer/ttl) that defines a TTL rule in the column's DDL. For example: `ttl: ts + INTERVAL 1 DAY` will be compiled as `TTL ts + INTERVAL 1 DAY`. |

### Example

```yaml
models:
- name: table_column_configs
description: 'Testing column-level configurations'
config:
contract:
enforced: true
columns:
- name: ts
data_type: timestamp
codec: ZSTD
- name: x
data_type: UInt8
ttl: ts + INTERVAL 1 DAY
```

## ClickHouse Cluster

Expand Down
6 changes: 5 additions & 1 deletion dbt/adapters/clickhouse/impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -545,8 +545,12 @@ def render_raw_columns_constraints(cls, raw_columns: Dict[str, Dict[str, Any]])
rendered_columns = []
for v in raw_columns.values():
codec = f"CODEC({_codec})" if (_codec := v.get('codec')) else ""
ttl = f"TTL {ttl}" if (ttl := v.get('ttl')) else ""
# Codec and TTL are optional clauses. The adapter should support scenarios where one
# or both are omitted. If specified together, the codec clause should appear first.
clauses = " ".join(filter(None, [codec, ttl]))
rendered_columns.append(
f"{quote_identifier(v['name'])} {v['data_type']} {codec}".rstrip()
f"{quote_identifier(v['name'])} {v['data_type']} {clauses}".rstrip()
)
if v.get("constraints"):
warn_or_error(ConstraintNotSupported(constraint='column', adapter='clickhouse'))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -256,3 +256,35 @@
1::Int32 as id,
toDate('2019-01-01') as date_day
"""

custom_constraint_model_schema_yml = """
version: 2
models:
- name: custom_column_constraint_model
materialized: table
config:
contract:
enforced: true
columns:
- name: id
data_type: Int32
codec: ZSTD
- name: ts
data_type: timestamp
- name: col_ttl
data_type: String
ttl: ts + INTERVAL 1 DAY
"""

check_custom_constraints_model_sql = """
{{
config(
materialized = "table",
)
}}

select
101::Int32 as id,
timestamp('2025-04-16') as ts,
'blue' as col_ttl
"""
17 changes: 16 additions & 1 deletion tests/integration/adapter/constraints/test_constraints.py
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
import pytest
from dbt.tests.util import get_manifest, run_dbt, run_dbt_and_capture, write_file
from fixtures_contraints import (

from tests.integration.adapter.constraints.fixtures_constraints import (
bad_column_constraint_model_sql,
bad_foreign_key_model_sql,
check_constraints_model_fail_sql,
check_constraints_model_sql,
check_custom_constraints_model_sql,
constraint_model_schema_yml,
contract_model_schema_yml,
custom_constraint_model_schema_yml,
model_data_type_schema_yml,
my_model_data_type_sql,
my_model_incremental_wrong_name_sql,
Expand Down Expand Up @@ -188,3 +191,15 @@ def test_model_constraints_fail_ddl(self, project):
["run", "-s", "check_constraints_model"], expect_pass=False
)
assert 'violated' in log_output.lower()


class TestModelCustomConstraints:
@pytest.fixture(scope="class")
def models(self):
return {
"check_custom_constraints_model.sql": check_custom_constraints_model_sql,
"constraints_schema.yml": custom_constraint_model_schema_yml,
}

def test_model_constraints_ddl(self, project):
run_dbt(["run", "-s", "check_custom_constraints_model"])