[Feature request][V2sink-StarRocks/Doris] Ensure exactly-once semantics

Search before asking

• I had searched in the feature and found no similar feature requirement.

Description

To ensure accurate one-time semantics for reading data sources to write StarRocks/Doris(or any other data destination database for this case), a policy can be adopted that writes only when Seatunnel is checkpoint;

There is a question about the existing configuration of StarRocks sink. Seatunnel ensures exactly once by checkpoint, but StarRocks sink can be configured with batch_max_rows, batch_max_bytes, batch_interval_ms and other parameters; If both exist at the same time, it cannot be guaranteed exactly once；

This is even better: when the exactly-once option is explicitly configured, Seatunnel is written if and only if it is checkpoint, and the batch_* configuration in the sink section should be invalidated automatically even if configured

Usage Scenario

增量读取jdbc，写入StarRocks

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[EricJoy2048]

I found some pull request about this feature #3850. Only write data when checkpoint is a way to ensure exactly-once. But I'm afraid this will cause OOM exception, because this may let sink save more data in memory. Some times the source read is very fast(such as File Source Connector), If the sink save data in memory, OOM is almost inevitable.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs.

[github-actions]

This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future.