-
Notifications
You must be signed in to change notification settings - Fork 1k
Description
Is your feature request related to a problem? Please describe.
The only way to rekey a topic is to use streams
Streams drop tombstones
There is this workaround https://rmoff.net/2020/11/03/kafka-connect-ksqldb-and-kafka-tombstone-messages/ that creates two independent streams but being independent they will presumably suffer from race conditions where one stream could be ahead of the other and a sequence of delete and adds could ultimately end in the wrong final state
Describe the solution you'd like
We need to be able to support rekeying and ideally other lightweight transformations with tombstones
Describe alternatives you've considered
Duplicate streams one handling the normal case and another handling the tonbstone
A sequence of new record followed by tombstone either stream could process the entire stream first resuting in both possible final states.
Additional context
processing everthing with tables beacuse they support tombstones uses more memory and has more overhead than a stimple stream.