clickhouse: fix date overflow for initial load by dtunikov · Pull Request #4534 · PeerDB-io/peerdb

dtunikov · 2026-07-03T14:48:48Z

clickhouse date types we currently use have min/max constraints (1900-2299)
this PR adds rounding logic for the dates outside this range

dtunikov · 2026-07-03T14:49:32Z

+	// Timestamps are micros since epoch; dates are days since epoch. Both encoded as Avro varint.
+	avroVal := t.UnixMicro()
+	if isDate {
+		avroVal = t.Unix() / 86400


had to move it here out of process[type] functions because we clamp t value for clickhouse

claude · 2026-07-03T14:57:13Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

codecov · 2026-07-03T15:20:33Z

⚠️ JUnit XML file not found

The CLI was unable to find any JUnit XML files to upload.
For more help, visit our troubleshooting guide.

github-actions · 2026-07-03T15:23:31Z

❌ Test Failure

Analysis: A deterministic assertion failure in Test_MySQL_DateTime_ClickHouse_Range (expected nil but got 1900-01-01) reproduces identically across all MySQL/MariaDB matrix variants, indicating the PR's own date-overflow fix is incomplete rather than a flaky failure.
Confidence: 0.97

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-03T15:42:30Z

❌ Test Failure

Analysis: Real bug: the PR's own test Test_MySQL_DateTime_ClickHouse_Range deterministically fails across all 4 CH suites and both matrix jobs with a clear value mismatch (out-of-range nullable DATE replicated as 1900-01-01 instead of NULL), with no timeout/race/network signals.
Confidence: 0.97

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

parseDateTime64BestEffortOrNull (used by the CDC normalize path) saturates out-of-range values to ClickHouse's [1900, 2299] boundary, clamping the date while preserving the time-of-day, and never nulls. Make the snapshot/Avro path do the exact same thing so snapshot and CDC stay consistent for the same source value, regardless of column nullability. Verified the Go clamp matches parseDateTime64BestEffortOrNull output exactly. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

github-actions · 2026-07-03T16:00:47Z

❌ Test Failure

Analysis: The PR's own new test Test_MySQL_DateTime_ClickHouse_Range fails deterministically with a logic assertion ("expected out-of-range nullable DATE to be NULL") across all 4 ClickHouse suite variants and all 3 matrix jobs, indicating the date-overflow fix is genuinely broken rather than flaky.
Confidence: 0.97

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-03T16:15:34Z

❌ Test Failure

Analysis: Real bug: Test_MySQL_DateTime_ClickHouse_Range fails deterministically across every matrix leg with an identical out-of-range datetime overflow (expected 2299-12-31 but got wrapped-around 1715-06-12), indicating a MySQL→ClickHouse datetime range-boundary conversion defect, not a flaky/timeout/race failure.
Confidence: 0.95

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

…ysql-datetime-date-clickhouse-range

The clickhouse-go driver decodes DateTime64 into a Go time.Time via int64 nanoseconds, which overflows near the 2299 boundary (2299-12-31 wraps to 1715-06-12). The clamped values are stored correctly; compare ClickHouse's own string rendering instead of scanning into time.Time. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

github-actions · 2026-07-06T09:35:39Z

❌ Test Failure

Analysis: Test_MySQL_DateTime_ClickHouse_Range failed deterministically across all 4 ClickHouse suites with an identical assertion (expected "1900-01-01", got ""), a real data-correctness bug in the exact date-overflow feature this PR modifies — not a flaky/timeout/race/network failure.
Confidence: 0.96

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-06T09:35:57Z

❌ Test Failure

Analysis: The PR's own feature test Test_MySQL_DateTime_ClickHouse_Range deterministically fails across all 8 MySQL/MariaDB×ClickHouse variants with expected "1900-01-01" but actual "", indicating a real bug in the date-overflow initial-load fix rather than a flaky failure.
Confidence: 0.96

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-06T09:39:04Z

❌ Test Failure

Analysis: The PR's own new test Test_MySQL_DateTime_ClickHouse_Range fails deterministically across four suites (expected "1900-01-01", got ""), a real bug in the datetime/date ClickHouse-range feature this branch implements; a concurrent BinlogIncident handshake-EOF failure looks flaky but is secondary.
Confidence: 0.9

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-06T09:55:10Z

❌ Test Failure

Analysis: Real bug: Test_MySQL_DateTime_ClickHouse_Range fails deterministically across all four suite variants with a data mismatch (expected "1900-01-01", got empty), directly tied to this PR's date-overflow fix for initial load.
Confidence: 0.97

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

Aliasing toString(col) back to the column's own name makes older ClickHouse (LTS) resolve the expression against the alias and return empty; the test only reads columns positionally, so the aliases are unnecessary. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

github-actions · 2026-07-06T10:26:14Z

❌ Test Failure

Analysis: Real bug: the PR's own test Test_MySQL_DateTime_ClickHouse_Range fails deterministically across all four ClickHouse+MySQL/MariaDB variants at the same assertion (clickhouse_mysql_test.go:1822) with a data mismatch (expected "1900-01-01"), indicating the date-overflow fix is incomplete rather than a flaky/timeout/race issue.
Confidence: 0.95

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-06T11:25:34Z

🔄 Flaky Test Detected

Analysis: A single matrix job failed at the "Install ClickHouse" setup step with a transient tarball download/decompress error (curl piped a non-gzip response into tar), which is a network flake unrelated to any test or code change.
Confidence: 0.97

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-07-06T11:28:36Z

🔄 Flaky Test Detected

Analysis: The job failed in the "Install ClickHouse" setup step (not in any test) when the ClickHouse binary download returned a non-gzip response causing tar to fail — a transient network/download flake, with the downstream ingest-results error being a mere cascade.
Confidence: 0.95

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-07-06T11:31:29Z

🔄 Flaky Test Detected

Analysis: The job failed in the "Install ClickHouse" setup step when a curl download of the ClickHouse tarball returned non-gzip content ("gzip: stdin: not in gzip format"), a transient network/CDN flake — no tests actually ran.
Confidence: 0.97

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-07-06T11:34:56Z

🔄 Flaky Test Detected

Analysis: The failure is a transient network flake in the "Install ClickHouse" setup step — the ClickHouse release tarball download returned non-gzip content ("gzip: stdin: not in gzip format"), aborting setup before any tests ran, so it's infrastructure flakiness rather than a real code bug.
Confidence: 0.95

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-07-06T11:38:10Z

🔄 Flaky Test Detected

Analysis: The job failed while downloading the ClickHouse tarball in the "Install ClickHouse" setup step ("gzip: stdin: not in gzip format" — a transient network/download error), not on any test, so no tests ran and the result is a CI infrastructure flake unrelated to the code change.
Confidence: 0.97

✅ Automatically retrying the workflow

View workflow run

dtunikov · 2026-07-06T11:44:28Z

        run: |
          if [ "${{ matrix.db-version.ch }}" = "lts" ]; then
-            echo "ch_version=v25.8.11.66-lts" >> $GITHUB_OUTPUT
+            echo "ch_version=v25.8.28.1-lts" >> $GITHUB_OUTPUT


it was consistently failing on v25.8.11.66-lts: https://github.com/PeerDB-io/peerdb/actions/runs/28783586496/job/85344784176

clickhouse_mysql_test.go:1822: Error Trace: /home/runner/work/peerdb/peerdb/flow/e2e/clickhouse_mysql_test.go:1822 /home/runner/work/peerdb/peerdb/flow/e2e/clickhouse_mysql_test.go:1828 /opt/hostedtoolcache/go/1.26.4/x64/src/runtime/asm_amd64.s:1771 Error: Not equal: expected: "1900-01-01" actual : ""

and what is weird is that clickhouse holds correct value
but clickhouse-go driver seems to interpret it incorrectly...
v25.8.28.1 is the latest patch of the oldest supported release (clickhouse LTS support seems to be 1 year).
so, I think it's ok if we test on the patched version, right?
i can also investigate further why v25.8.11.66 wasn't working properly, but it seems out-of-scope as it works fine on patched version.

Seems niche enough, yeah

fix clickhouse date overflow

ebb0355

dtunikov requested a review from a team as a code owner July 3, 2026 14:48

dtunikov commented Jul 3, 2026

View reviewed changes

default to unix epoch when we hit date overflow for ch

204ab19

dtunikov marked this pull request as draft July 3, 2026 15:05

use slog.DiscardHandler in test to satisfy sloglint

1b13de0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

cleanup comments

e89439c

dtunikov and others added 4 commits July 6, 2026 10:52

Merge branch 'main' of github.com:PeerDB-io/peerdb into fix/dbi-812/m…

1d88b80

…ysql-datetime-date-clickhouse-range

cleanup comments

1dcb3bc

rm unit test

8701ccc

dtunikov marked this pull request as ready for review July 6, 2026 09:15

avoid string concat in loop (perfsprint)

24b996c

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

move ch CI versions

cebfe42

stable tag

de08694

dtunikov commented Jul 6, 2026

View reviewed changes

jgao54 approved these changes Jul 6, 2026

View reviewed changes

ilidemi approved these changes Jul 7, 2026

View reviewed changes

dtunikov merged commit c2a2c33 into main Jul 7, 2026
21 checks passed

dtunikov deleted the fix/dbi-812/mysql-datetime-date-clickhouse-range branch July 7, 2026 07:58

Uh oh!

Conversation

dtunikov commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dtunikov Jul 3, 2026

Choose a reason for hiding this comment

Uh oh!

claude Bot commented Jul 3, 2026

Code review

Uh oh!

codecov Bot commented Jul 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

⚠️ JUnit XML file not found

Uh oh!

github-actions Bot commented Jul 3, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 3, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 3, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 3, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 6, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 6, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 6, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 6, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 6, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 6, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Jul 6, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Jul 6, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Jul 6, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Jul 6, 2026

🔄 Flaky Test Detected

Uh oh!

dtunikov Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilidemi Jul 7, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dtunikov commented Jul 3, 2026 •

edited

Loading

codecov Bot commented Jul 3, 2026 •

edited

Loading

dtunikov Jul 6, 2026 •

edited

Loading