maria: classify compressed columns errors + add validation for create mirror by dtunikov · Pull Request #4559 · PeerDB-io/peerdb

dtunikov · 2026-07-09T15:12:01Z

mariadb has COMPRESSED column feature
go-mysql can't decode such columns
now we classify if such column is added mid cdc stream
we also validate the source and error out if there are compressed columns

Copilot

Pull request overview

Adds first-class handling for MariaDB COMPRESSED columns, which go-mysql cannot decode, by (a) rejecting mirrors at validation time when such columns exist and (b) classifying/raising a dedicated error if a compressed column appears mid-CDC via TABLE_MAP detection.

Changes:

Introduces MySQLUnsupportedCompressedColumnError and wires it into the alerting classifier for user notification.
Adds MariaDB source validation that scans table schemas for COMPRESSED columns and rejects mirror creation.
Adds CDC-time detection (TABLE_MAP wire types 140/141) plus unit/e2e coverage for both “rejected at creation” and “added mid-stream” scenarios.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
flow/shared/exceptions/mysql.go	Adds a typed error for unsupported MariaDB COMPRESSED columns.
flow/pkg/mysql/validation.go	Adds `IsCompressedColumnType` + table-level validation query for COMPRESSED columns.
flow/pkg/mysql/validation_test.go	Unit tests for COMPRESSED column type detection helper.
flow/e2e/clickhouse_mysql_test.go	E2E coverage for mirror validation rejection + mid-CDC detection.
flow/connectors/mysql/validate.go	Hooks COMPRESSED-column validation into mirror source validation for MariaDB flavor.
flow/connectors/mysql/type_conversion.go	Errors out early when a column type string indicates COMPRESSED.
flow/connectors/mysql/qvalue_convert_test.go	Unit test ensuring type conversion rejects COMPRESSED.
flow/connectors/mysql/cdc.go	Detects COMPRESSED wire types in TABLE_MAP events and returns typed exception.
flow/connectors/mysql/cdc_test.go	Unit tests for TABLE_MAP compressed-column detection + typed error contents.
flow/alerting/classifier.go	Classifies the new typed exception into a NotifyUser error class.
flow/alerting/classifier_test.go	Tests classifier mapping for the new COMPRESSED-column exception.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

claude · 2026-07-09T15:18:24Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

github-actions · 2026-07-09T15:39:54Z

❌ Test Failure

Analysis: Every MariaDB CDC e2e test times out waiting for CDC/snapshot data — reproduced deterministically across all three matrix jobs on this MariaDB-focused PR — indicating a real bug the PR introduced in the MariaDB pipeline, not a flaky failure.
Confidence: 0.93

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-07-09T16:40:41Z

🔄 Flaky Test Detected

Analysis: The flow/e2e package hit its 20-minute wall-clock timeout with many unrelated ClickHouse tests still stuck in EnvWaitFor CDC-sync polling loops — a classic e2e timeout flake from a slow/contended runner, not an assertion failure or a bug in the MariaDB compressed-columns PR.
Confidence: 0.75

✅ Automatically retrying the workflow

View workflow run

github-actions · 2026-07-09T17:05:35Z

🔄 Possible Flaky Test

Analysis: All three flow/e2e matrix jobs hit the 20-minute package wall-clock timeout (1200.9s) while every in-flight test showed healthy short durations (3s–1m7s) with no test hung in EnvWaitFor, indicating the heavy e2e suite narrowly exceeded its time budget under slow/contended CI rather than a specific test failing or deadlocking.
Confidence: 0.6

⚠️ Confidence too low (0.6) to retry automatically - manual review recommended

View workflow run

github-actions · 2026-07-09T17:07:56Z

❌ Test Failure

Analysis: All three MySQL/MariaDB e2e matrix variants consistently hung to the 20-minute package timeout (panic: test timed out, no assertion failure) in exactly the area this PR modifies, indicating a real hang/deadlock rather than a flaky failure.
Confidence: 0.82

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

…db-compressed-columns

…B-io/peerdb into dbi-847-mariadb-compressed-columns

codecov · 2026-07-09T18:13:36Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
3033	2	3031	400

View the top 2 failed test(s) by shortest run time

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH

Stack Traces | 0.02s run time

=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH (0.02s)

github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_CompressedColumn_AddedMidCDC

Stack Traces | 204s run time

=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_CompressedColumn_AddedMidCDC
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_CompressedColumn_AddedMidCDC
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_CompressedColumn_AddedMidCDC
2026/07/10 15:30:02 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/10 15:30:02 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/10 15:30:02 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_ewnanhef.test_mysql_schema_changes
    clickhouse_mysql_test.go:243: WaitFor waiting on initial snapshot 2026-07-10 15:30:10.379046203 +0000 UTC m=+918.562515943
2026/07/10 15:30:10 INFO Received AWS credentials from peer for connector: clickhouse x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
    clickhouse_mysql_test.go:248: WaitFor waiting for CDC to start 2026-07-10 15:30:10.383861893 +0000 UTC m=+918.567331623
    clickhouse_mysql_test.go:257: WaitFor waiting for compressed column error 2026-07-10 15:30:25.455093561 +0000 UTC m=+933.638563312
2026/07/10 15:30:25 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mychcl_ukleki7d.test_unsigned
2026/07/10 15:30:25 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_0rn9fw9j.test_add_col_types
    clickhouse_mysql_test.go:257: UNEXPECTED TIMEOUT waiting for compressed column error 2026-07-10 15:33:25.574624655 +0000 UTC m=+1113.758094395
    clickhouse_mysql_test.go:236: flow log after test failure: type=info message=Setting up destination tables
    clickhouse_mysql_test.go:236: flow log after test failure: type=info message=created table compressed_mid_cdc in destination
    clickhouse_mysql_test.go:236: flow log after test failure: type=info message=All destination tables have been setup
    clickhouse_mysql_test.go:236: flow log after test failure: type=info message=Setting up replication slot and publication
    clickhouse_mysql_test.go:236: flow log after test failure: type=info message=stored 1 records into intermediate storage for batch 1 in 15s
    clickhouse_mysql_test.go:236: flow log after test failure: type=error message=MySQL execute error: Header &replication.EventHeader{Timestamp:0x6a511011, EventType:0x17, ServerID:0x1, EventSize:0x3e, LogPos:0x0, Flags:0x0}, Data "\xd8\x00\x00\x00\x00\x00\x01\x00\x03\a\xf8\x03\x00\x00\x00\x03\x00cdc\x11\x00\x00compressed value", Err: unsupport type 140 in binlog and don't know how to handle
    clickhouse_mysql_test.go:236: flow log after test failure: type=error message=MySQL execute error: Header &replication.EventHeader{Timestamp:0x6a511011, EventType:0x17, ServerID:0x1, EventSize:0x3e, LogPos:0x0, Flags:0x0}, Data "\xd8\x00\x00\x00\x00\x00\x01\x00\x03\a\xf8\x03\x00\x00\x00\x03\x00cdc\x11\x00\x00compressed value", Err: unsupport type 140 in binlog and don't know how to handle
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_CompressedColumn_AddedMidCDC (203.62s)

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

github-actions · 2026-07-09T18:17:54Z

❌ Test Failure

Analysis: The newly-added Test_MariaDB_CompressedColumn_AddedMidCDC fails deterministically across all three MariaDB_CH matrix configs by timing out waiting for the PR's own new compressed-column error to appear, indicating a real bug in the feature under test rather than flakiness.
Confidence: 0.9

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-09T18:19:51Z

❌ Test Failure

Analysis: All 3 MySQL matrix jobs hit a 20-minute e2e package timeout with only MySQL tests stuck in EnvWaitFor (replication never delivered rows), a deterministic MySQL-specific replication stall on a PR that specifically changes MySQL/MariaDB handling — consistent with a real regression, not flakiness.
Confidence: 0.78

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-10T15:08:17Z

❌ Test Failure

Analysis: The failure is a WaitFor timeout in Test_MariaDB_CompressedColumn_AddedMidCDC, a brand-new test added by this PR to verify its own compressed-column handling feature, which never produced the expected snapshot/error behavior — a real functional bug rather than an intermittent infra failure.
Confidence: 0.8

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-10T15:14:05Z

❌ Test Failure

Analysis: The new feature test Test_MariaDB_CompressedColumn_AddedMidCDC deterministically fails because MariaDB's compressed-column binlog type (140) is unhandled ("unsupport type 140 in binlog"), so the expected compressed-column-error condition never occurs — a real bug in the PR's feature, not flakiness.
Confidence: 0.9

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

github-actions · 2026-07-10T15:36:00Z

❌ Test Failure

Analysis: The PR's own new test Test_MariaDB_CompressedColumn_AddedMidCDC fails deterministically across multiple matrix legs because the code emits an unhandled "unsupport type 140 in binlog" error instead of the classified compressed-column error the test expects — a real feature bug, not flakiness.
Confidence: 0.92

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

classify compressed columns errors + add validation for create mirror

b872fbb

dtunikov requested a review from Copilot July 9, 2026 15:12

dtunikov requested a review from a team as a code owner July 9, 2026 15:12

Copilot started reviewing on behalf of dtunikov July 9, 2026 15:12 View session

Copilot AI reviewed Jul 9, 2026

View reviewed changes

Comment thread flow/connectors/mysql/type_conversion.go

Comment thread flow/e2e/clickhouse_mysql_test.go

Potential fix for pull request finding

ff37778

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

dtunikov added 4 commits July 9, 2026 19:47

fix type enum

cf6f251

Merge branch 'main' of github.com:PeerDB-io/peerdb into dbi-847-maria…

c5d1c6f

…db-compressed-columns

Merge branch 'dbi-847-mariadb-compressed-columns' of github.com:PeerD…

e900e32

…B-io/peerdb into dbi-847-mariadb-compressed-columns

fix e2e test

6bcc5a0

dtunikov added 2 commits July 10, 2026 16:39

try waiting 6 min

3ee3427

fix tests

4b9ceb4

update test comment

88f1277

Uh oh!

Conversation

dtunikov commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

claude Bot commented Jul 9, 2026

Code review

Uh oh!

github-actions Bot commented Jul 9, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 9, 2026

🔄 Flaky Test Detected

Uh oh!

github-actions Bot commented Jul 9, 2026

🔄 Possible Flaky Test

Uh oh!

github-actions Bot commented Jul 9, 2026

❌ Test Failure

Uh oh!

codecov Bot commented Jul 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

github-actions Bot commented Jul 9, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 9, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 10, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 10, 2026

❌ Test Failure

Uh oh!

github-actions Bot commented Jul 10, 2026

❌ Test Failure

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

dtunikov commented Jul 9, 2026 •

edited

Loading

codecov Bot commented Jul 9, 2026 •

edited

Loading