[FLINK-38691] [cdc connector mysql] Support for MySQL Transaction Boundary Events in Flink CDC Connector#4170
Merged
Merged
Conversation
tejanshrana
changed the title
ruanhang1993
approved these changes
Jan 26, 2026
Contributor
ruanhang1993
left a comment
ruanhang1993
left a comment
There was a problem hiding this comment.
@tejanshrana Thanks for the PR.
LGTM totally. @lvyanquan Could you help to review and see if it could be included in 3.6 version?
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request adds support for MySQL transaction boundary events (BEGIN/END) in the Flink CDC MySQL connector. Previously, when Debezium's provide.transaction.metadata was enabled, transaction metadata events were logged as "unknown" and skipped, causing loss of transaction boundary information.
Changes:
- Added detection logic for transaction metadata events using the Debezium schema name pattern
- Transaction events now update binlog offsets regardless of configuration
- Added optional emission of transaction events controlled by
includeTransactionMetadataEventsflag (default: false)
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| RecordUtils.java | Added constant and detection method for transaction metadata events using keySchema pattern |
| MySqlRecordEmitter.java | Added conditional processing and emission logic for transaction metadata events |
| MySqlSourceConfig.java | Added includeTransactionMetadataEvents configuration field |
| MySqlSourceConfigFactory.java | Added builder method and Debezium property configuration for transaction metadata |
| MySqlSourceBuilder.java | Added public API method to configure transaction metadata event inclusion |
| MySqlSource.java | Updated MySqlRecordEmitter instantiation to pass transaction metadata flag |
| MySqlPipelineRecordEmitter.java | Explicitly disabled transaction metadata events for pipeline connector |
| MySqlRecordEmitterTest.java | Added comprehensive test coverage for transaction metadata event handling |
| MySqlSourceReaderTest.java | Updated test fixtures to pass new constructor parameter |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
lvyanquan
approved these changes
Jan 27, 2026
Contributor
lvyanquan
left a comment
lvyanquan
left a comment
There was a problem hiding this comment.
LGTM, left one minor suggestion.
Contributor
|
Thanks @tejanshrana's contribution, merged. |
Mrart
pushed a commit
to Mrart/flink-cdc
that referenced
this pull request
Mar 26, 2026
…ndary Events in Flink CDC Connector (apache#4170)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Transaction BEGIN/END events (when provide.transaction.metadata is enabled) are currently treated as unknown and skipped, causing loss of transaction boundaries.
Solution
Logs from before the change:
2025-11-04 14:52:28 2025-11-04 14:52:28,827 INFO org.apache.flink.cdc.connectors.mysql.source.reader.MySqlRecordEmitter [] - Meet unknown element SourceRecord{sourcePartition={server=mysql_binlog_source}, sourceOffset={transaction_id=4541146d-b988-11f0-87f6-0242ac140006:33, ts_sec=1762267948, file=mysql-bin.000003, pos=13381, gtids=4541146d-b988-11f0-87f6-0242ac140006:1-32, server_id=1}} ConnectRecord{topic='mysql_binlog_source.transaction', kafkaPartition=null, key=Struct{id=4541146d-b988-11f0-87f6-0242ac140006:33}, keySchema=Schema{io.debezium.connector.common.TransactionMetadataKey:STRUCT}, value=Struct{status=BEGIN,id=4541146d-b988-11f0-87f6-0242ac140006:33}, valueSchema=Schema{io.debezium.connector.common.TransactionMetadataValue:STRUCT}, timestamp=null, headers=ConnectHeaders(headers=)}, just skip.After the change, the transaction start and end events look like below:
{"status":"BEGIN","id":"4541146d-b988-11f0-87f6-0242ac140006:47","event_count":null,"data_collections":null} {"status":"END","id":"4541146d-b988-11f0-87f6-0242ac140006:47","event_count":1,"data_collections":[{"data_collection":"sample.sample","event_count":1}]}