JSONL Source Connector Configuration #558
Description
Hello!
I would like to publish messages out of a bucket using JSONL. It seems like the sink connector produces a message of the following format based on the documentation:
{
"key": ...,
"value": { ... }
}
I would like to publish Kafka messages with the key field being the Kafka Message Key, and the value field be the Kafka Message Value.
I am trying to have a file with multiple messages that I can then publish using this format, but I am running into issues because of the native key and (I'm assuming) schema-less parsing of JSON messages. Based on this line:
It seems like if JSONL is the input.format, the produced Kafka Connect Record will be without a schema. This leads to issues if you try to apply SMTs to the message to extract the key and value of the message using transforms like ValueToKey and ExtractField to use the key field as the message key, and using ExtractField on value to unwrap the JSON payload for the message value.
Is there another approach that would allow me to do this? Thanks!