Kafka Druid ingestion: only the first key field ingested and others ignored? #18578
Description
We’re trying to set up Kafka ingestion in Druid so that all fields from both the Kafka key and value are included in the datasource.
Right now, only the first field of the key is being ingested. I found this note in the documentation:
“The input format to parse the Kafka key only processes the first entry of the inputFormat field. If your key values are simple strings, you can use the tsv format to parse them. Note that for tsv, csv, and regex formats, you need to provide a columns array to make a valid input format. Only the first one is used, and its name will be ignored in favor of keyColumnName.”
Did I miss something?
Is there a way to ingest all Kafka key fields as columns in the Druid datasource—without copying them into the value?