You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/content/docs/connectors/flink-sources/mysql-cdc.md
+24Lines changed: 24 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -874,6 +874,30 @@ There are two places that need to be taken care of.
874
874
*If no update operation is performed on the specified column, the exactly-once semantics is ensured.
875
875
*If the update operation is performed on the specified column, only the at-least-once semantics is ensured. However, you can specify primary keys at downstream and perform the idempotence operation to ensure data correctness.
876
876
877
+
#### Warning
878
+
879
+
Using a **non-primary key column** as the `scan.incremental.snapshot.chunk.key-column` for a MySQL table with primary keys may lead to data inconsistencies. Below is a scenario illustrating this issue and recommendations to mitigate potential problems.
880
+
881
+
#### ProblemScenario
882
+
883
+
-**TableStructure:**
884
+
-**PrimaryKey:** `id`
885
+
-**ChunkKeyColumn:** `pid` (Not a primary key)
886
+
887
+
-**SnapshotSplits:**
888
+
-**Split0:** `1< pid <=3`
889
+
-**Split1:** `3< pid <=5`
890
+
891
+
-**Operation:**
892
+
-Two different subtasks are reading Split0 and Split1 concurrently.
893
+
-An update operation changes `pid` from `2` to `4` for `id=0` while both splits are being read. This update occurs between the low and high watermark of both splits.
894
+
895
+
-**Result:**
896
+
-**Split0:**Contains the record `[id=0, pid=2]`
897
+
-**Split1:**Contains the record `[id=0, pid=4]`
898
+
899
+
Since the order of processing these records cannot be guaranteed, the final value of `pid` for `id=0` may end up being either `2` or `4`, leading to potential data inconsistencies.
900
+
877
901
### About converting binary type data to base64 encoded data
Copy file name to clipboardExpand all lines: flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/debezium/reader/BinlogSplitReaderTest.java
Copy file name to clipboardExpand all lines: flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/MySqlSourceITCase.java
Copy file name to clipboardExpand all lines: flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/reader/MySqlSourceReaderTest.java
Copy file name to clipboardExpand all lines: flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/source/utils/SnapshotFilterUtilsTest.java
Copy file name to clipboardExpand all lines: flink-cdc-connect/flink-cdc-source-connectors/flink-connector-mysql-cdc/src/test/java/org/apache/flink/cdc/connectors/mysql/table/MySqlTableSourceFactoryTest.java
0 commit comments