Commit 1590ddb
### Description
This PR resolves a raw staging table creation crash
(`BAD_TTL_EXPRESSION`) on older ClickHouse versions/clusters
(specifically ClickHouse version <= 24.2).
#### The Problem
In older ClickHouse clusters, the database engine expects `DateTime` or
`Date` columns/expressions inside a table's `TTL` definition. PeerDB's
current staging table implementation uses
`fromUnixTimestamp64Nano(_peerdb_timestamp)` which yields a
high-precision `DateTime64(9)`. This triggers a `BAD_TTL_EXPRESSION`
error, blocking schema replaying and staging.
#### The Fix
This PR wraps `fromUnixTimestamp64Nano(_peerdb_timestamp)` in a
`toDateTime()` cast, which truncates the nanosecond precision to
seconds. Since staging tables are built to retain CDC logs over a course
of days (defaulting to 90 days), this second-level precision has zero
functional impact on data eviction accuracy but ensures compatibility
with older ClickHouse clusters while continuing to work perfectly on
newer versions.
### Changes
* **CDC DDL Generation**: Modified raw table creation in
`flow/connectors/clickhouse/cdc.go` to use
`toDateTime(fromUnixTimestamp64Nano(_peerdb_timestamp))`.
* **CDC Tests**: Updated assertions in
`flow/connectors/clickhouse/cdc_test.go` to match the new `toDateTime`
format.
* **Normalize Query Tests**: Modified
`flow/connectors/clickhouse/normalize_query_test.go` to dynamically
fetch the ClickHouse native test port and host from the environment
helper instead of hardcoding `localhost:9000`.
* **Dev Environment**: Simplified the `env_file` formatting in
`docker-compose-dev.yml` to be backward-compatible with Docker Compose
versions <= v2.13.
### Verification Results
All tests in the ClickHouse connector package pass successfully locally
against the test catalog and ClickHouse instance:
```bash
cd flow
PEERDB_CATALOG_HOST=localhost CI_CLICKHOUSE_HOST=localhost go test -v ./connectors/clickhouse/...
---------
Co-authored-by: Ilia Demianenko <belk94@gmail.com>
1 parent f791216 commit 1590ddb
3 files changed
Lines changed: 5 additions & 2 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
77 | 77 | | |
78 | 78 | | |
79 | 79 | | |
80 | | - | |
| 80 | + | |
81 | 81 | | |
82 | 82 | | |
83 | 83 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
132 | 132 | | |
133 | 133 | | |
134 | 134 | | |
| 135 | + | |
135 | 136 | | |
136 | 137 | | |
137 | 138 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
12 | 13 | | |
13 | 14 | | |
14 | 15 | | |
15 | 16 | | |
16 | 17 | | |
17 | | - | |
| 18 | + | |
| 19 | + | |
18 | 20 | | |
19 | 21 | | |
20 | 22 | | |
| |||
0 commit comments