Skip to content

maria: require resync on partial row events#4530

Merged
dtunikov merged 17 commits into
mainfrom
fix/dbi-815/mysql-fail-on-maria-partial-row-data-event
Jul 8, 2026
Merged

maria: require resync on partial row events#4530
dtunikov merged 17 commits into
mainfrom
fix/dbi-815/mysql-fail-on-maria-partial-row-data-event

Conversation

@dtunikov

@dtunikov dtunikov commented Jul 2, 2026

Copy link
Copy Markdown
Collaborator

We don't support partial row events. MariaDB 12.3+ emits those when payload size is bigger than binlog_row_event_fragment_threshold (default 1Gb).
This PR handles this event type and requires a resync.
In addition, it introduces UnsupportedBinlogEventCounter(eventType) metric to keep track of skipped events.

@dtunikov dtunikov requested a review from a team as a code owner July 2, 2026 11:24
@claude

claude Bot commented Jul 2, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

- add metric to count unsupported event types
@dtunikov dtunikov changed the title fail loudly on partial row maria event maria: require resync on partial row events Jul 2, 2026
Comment thread flow/connectors/mysql/cdc.go Outdated
Comment thread flow/connectors/mysql/cdc.go Outdated
Comment thread flow/e2e/clickhouse_mysql_test.go
Comment thread flow/e2e/mysql.go Fixed
Comment thread flow/alerting/classifier.go Outdated
Comment thread flow/shared/exceptions/mysql.go
Comment thread flow/e2e/clickhouse_mysql_test.go Outdated

@jgao54 jgao54 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few nit comment but lgtm overall

Comment thread flow/connectors/mysql/cdc.go Outdated
Comment thread flow/e2e/clickhouse_mysql_test.go
Comment thread flow/shared/exceptions/mysql.go
Comment thread flow/connectors/mysql/cdc.go Outdated
@codecov

codecov Bot commented Jul 3, 2026

Copy link
Copy Markdown

❌ 2 Tests Failed:

Tests completed Failed Passed Skipped
2954 2 2952 343
View the top 1 failed test(s) by shortest run time
github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent
Stack Traces | 195s run time
=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent
2026/07/03 13:45:02 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_xsk2ix5z.test_mysql_schema_changes
    clickhouse_mysql_test.go:1957: WaitFor waiting for in-pipe row past out-of-pipe partial row event 2026-07-03 13:45:16.564518412 +0000 UTC m=+485.130611153
2026/07/03 13:45:16 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_1iwesbnb.test_blobs
2026/07/03 13:45:16 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_mach_6zelra5m.test_bit
    clickhouse_mysql_test.go:1957: UNEXPECTED TIMEOUT waiting for in-pipe row past out-of-pipe partial row event 2026-07-03 13:48:17.177314151 +0000 UTC m=+665.743406874
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH/Test_MariaDB_PartialRowEvent (194.78s)
View the full list of 1 ❄️ flaky test(s)
github.com/PeerDB-io/peerdb/flow/e2e::TestPeerFlowE2ETestSuiteMariaDB_CH

Flake rate in main: 14.29% (Passed 24 times, Failed 4 times)

Stack Traces | 0.01s run time
=== RUN   TestPeerFlowE2ETestSuiteMariaDB_CH
=== PAUSE TestPeerFlowE2ETestSuiteMariaDB_CH
=== CONT  TestPeerFlowE2ETestSuiteMariaDB_CH
--- FAIL: TestPeerFlowE2ETestSuiteMariaDB_CH (0.01s)
2026/07/03 13:49:14 INFO Received AWS credentials from peer for connector: ci x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN}
2026/07/03 13:49:14 INFO fetched schema x-peerdb-additional-metadata={Operation:FLOW_OPERATION_UNKNOWN} table=e2e_test_machcl_7ipudwcu.test_composite_pkey_ordering

To view more test analytics, go to the Test Analytics Dashboard
📋 Got 3 mins? Take this short survey to help us improve Test Analytics.

@github-actions

github-actions Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

❌ Test Failure

Analysis: The PR's own new test Test_MariaDB_PartialRowEvent fails deterministically across all three matrix jobs (mysql-gtid/pos, MySQL 6.0/7.0/8.0) after ~190s, indicating a real bug in the PR's partial-row-event resync logic rather than a flaky failure.
Confidence: 0.9

⚠️ This appears to be a real bug - manual intervention needed

View workflow run

Comment thread flow/connectors/mysql/cdc_test.go Outdated
Comment thread flow/connectors/mysql/cdc.go
Comment thread flow/alerting/classifier.go
Comment thread flow/connectors/mysql/cdc.go Outdated
Comment thread flow/otel_metrics/otel_manager.go Outdated
Comment thread flow/connectors/mysql/cdc.go
Comment thread flow/connectors/mysql/cdc_test.go Outdated
Comment thread flow/shared/exceptions/mysql.go
- add AdditionalAttributes to classified error
@dtunikov dtunikov merged commit 4d951f4 into main Jul 8, 2026
19 checks passed
@dtunikov dtunikov deleted the fix/dbi-815/mysql-fail-on-maria-partial-row-data-event branch July 8, 2026 08:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants