Input schema_registry_encoder does not handle kafka tombstones

[Pavelsky89]

Hi all

It looks like schema_registry_encoder does not handle kafka tombstones correctly, instead of passing downstream null, it fails with error

5 byte header for value is missing or does not have 0 magic byte

[rockwotj]

You can workaround this by using the branch processor and the metadata for if the record is a tombstone

[Pavelsky89]

Thanks for the tip but that does not work in my case, my internal pipeline might filter out some messages and in that case the branch processor fails with error "child processors resulted in zero messages"

[rockwotj]

You can move the filter outside of the branch or use a sentinel value to delete before or after the branch for example:

mutation: |
  if @kafka_tombstone {
    root = deleted()
  }

[Pavelsky89]

moving deletion outside of branch is not an option in my case, I was thinking about hack to mark deleted value using metadata header and delete it afterwards, that will probably work.

I have however (not tested yet) use case when some messages got unarchived into multiple messages, will message size increase be accepted by branch processor?