Skip to content

Streams supporting tombstones #10639

@sillencem

Description

@sillencem

Is your feature request related to a problem? Please describe.
The only way to rekey a topic is to use streams
Streams drop tombstones
There is this workaround https://rmoff.net/2020/11/03/kafka-connect-ksqldb-and-kafka-tombstone-messages/ that creates two independent streams but being independent they will presumably suffer from race conditions where one stream could be ahead of the other and a sequence of delete and adds could ultimately end in the wrong final state

Describe the solution you'd like
We need to be able to support rekeying and ideally other lightweight transformations with tombstones

Describe alternatives you've considered
Duplicate streams one handling the normal case and another handling the tonbstone
A sequence of new record followed by tombstone either stream could process the entire stream first resuting in both possible final states.

Additional context
processing everthing with tables beacuse they support tombstones uses more memory and has more overhead than a stimple stream.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions