Skip to content

Conversation

yzeng1618
Copy link

https://issues.apache.org/jira/browse/FLINK-34864
Previously, in the scenario of multi-table synchronization such as from MySQL to Doris, if some tables had no primary keys, it would directly result in an error, which affected the synchronization of other tables. Therefore, relevant development work was carried out. Incidentally, I found that this issue had already been recorded on Jira. As a result, I added unit tests and other content and submitted this pull request (PR). The specific usage is to set the ignoreNoPrimaryKeyTable parameter to "true" in the pipeline of the YAML file, and then this function can be achieved.

…rameter to skip tables without primary keys in multi-table synchronization
"Whether to ignore tables without primary key. When enabled, the connector will skip tables "
+ "that don't have a primary key. By default these tables will be processed, but for some "
+ "scenarios it may be desirable to ignore them since tables without primary keys "
+ "might cause performance issues during incremental snapshot.");
Copy link
Contributor

@lvyanquan lvyanquan May 27, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What would happen if we met the records of these tables in incremental reading phase? Skip it or emit it?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we will skip them

Copy link
Contributor

@lvyanquan lvyanquan May 29, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add an ITCase test to verify it during snapshot and incremental reading phases?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The code logic content has been newly added and verified through testing.

@github-actions github-actions bot added the docs Improvements or additions to documentation label May 28, 2025
@lvyanquan lvyanquan added the 3.5 label May 29, 2025
zengyi added 2 commits June 3, 2025 10:36
…c table creation of tables without primary keys and the issue of incremental data synchronization
# Conflicts:
#	docs/content.zh/docs/connectors/flink-sources/mysql-cdc.md
#	docs/content/docs/connectors/flink-sources/mysql-cdc.md
@yzeng1618 yzeng1618 requested a review from lvyanquan June 6, 2025 05:57
@yzeng1618
Copy link
Author

@lvyanquan Could you please let me know when you can take a look at this PR and merge it?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3.5 docs Improvements or additions to documentation mysql-cdc-connector mysql-pipeline-connector

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants