how to read incremental data from source table and sink to sink table using apache seatunnel?

[sangeetha1706]

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

i want to fetch incremental data from mysql source table based on last fetch id or last inserted timestamp and sink data to destination table.i should read data from source table where it stopped fetching last time and continue from the latest records which got inserted to sopurce table and sink that to destination table.how can i achieve this in apache seatunnel ?

Usage Scenario

No response

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[EricJoy2048]

You can use CDC connector like MySQL CDC Source Connector https://seatunnel.incubator.apache.org/docs/connector-v2/source/MySQL-CDC.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[YuriyGavrilov]

any other ideas? @EricJoy2048 there are not all connectors support cdc
There could be any sql based table or file. What is best pattern to offset already loaded or computed data in the source and sink only difference? some time there could be an issue or network blinking or technical maintain hours and need to be sure in when I turn on seatunnel it didn't load the full data again or will not making any overwork.