Bug: Duplicate records can be introduced when failure before commits happen

[lyuboxa]

Bug description

The source connector batch acker, will commit offsets to Kafka at every n or 1000 (as const).
This can result in failure introducing between 1 < duplicates < n to the destination.
When the consumer restarts, it will assume the consumer group contains the latest positioning data
ask'd by Conduit, however the discrepancy described above can be observed.

Steps to reproduce

1. Create kafka pipeline with conduit
2. Produce 10 records
3. Kill pipeline (kill command not teardown, as this will force a flush)
4. Start pipeline, same records will be produced

Version

latest

[lovromazgon]

@lyuboxa isn't this a duplicate of #220?

[lyuboxa]

@lyuboxa isn't this a duplicate of #220?

The issue was already presented in #220 but #231 didn't solve the issue with duplicates and I made this issue to track it.
This is not done, it will require some work to skip records until the positional offset saved in conduit is reached, which may differ from that of the consume group. This is the implication of failure occurring in between flushes with lost acks.

[lovromazgon]

I see, we can add this record skipping behavior, so we don't repeat records that we know we saw based on the position coming from Conduit.

That said, be aware that killing Conduit without a graceful teardown is already prone to produce duplicates, even if we add this record skipping logic. There is a time window when records are successfully written to the destination, but not yet acked in the source. If you manage to kill Conduit in that window, you'll see duplicates, and the connector can't do anything about it.

[lyuboxa]

That said, be aware that killing Conduit without a graceful teardown is already prone to produce duplicates, even if we add this record skipping logic. There is a time window when records are successfully written to the destination, but not yet acked in the source. If you manage to kill Conduit in that window, you'll see duplicates, and the connector can't do anything about it.

Yea, but that is a physics problem and somewhat expected. As far as I know we are not making exactly-once guarantees. But rather at-least-once