Request for CloudEvent serde

[supriya-albal-polestar]

Issue submitter TODO list

• I've searched for an already existing issues here
• I'm running a supported version of the application which is listed here and the feature is not present there

Is your proposal related to a problem?

No response

Describe the feature you're interested in

We use CloudEvent to wrap our kafka messages. As a result, we are seeing encoded values in Provectus, that we are unable to read.
Here is link to CloudEvent library - https://cloudevents.github.io/sdk-java/kafka.html.

Describe alternatives you've considered

No response

Version you're running

56fa824 v0.7.1

Additional context

No response

[github-actions]

Hello there supriya-albal-polestar! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

[supriya-albal-polestar]

I would like to get inputs on this issue.

[Haarolean]

Hi, we made the serde API for this exact purpose -- so the community can create pluggable SerDes.
Currently, we don't have any plans to develop any other serdes, especially the ones with narrow use-cases.
Feel free to reach out for professional services for Apache Kafka by Provectus if you wish to get one developed for you personally, though.

[supriya-albal-polestar]

@Haarolean Is there any documentation or example for using protobuf? I see that protoBuf is supported. Can I use proto object for CloudEvent with kafka-ui? I tried by referring to one yml example in your code, but it did not help.
Have also asked on stackoverflow - https://stackoverflow.com/questions/77077386/how-to-use-custom-serde-with-provectus-kafka-ui?noredirect=1#comment135879983_77077386

[Haarolean]

Here should be all the info on the protobuf serde.

[supriya-albal-polestar]

thanks @Haarolean

I referred to the documents and created following docker compose -

kafka:
image: confluentinc/cp-kafka:6.0.2
container_name: kafka
restart: always
depends_on:
- zookeeper
ports:
- "9092:9092"
- "29092:29092"
extra_hosts:
- "host.docker.internal:host-gateway"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

kafka-ui:
container_name: kafka-ui
image: provectuslabs/kafka-ui:latest
ports:
- "8080:8080"
depends_on:
- zookeeper
- kafka
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka:29092
KAFKA_CLUSTERS_0_ZOOKEEPER: zookeeper:2181
KAFKA_CLUSTERS_0_JMXPORT: 9997
kafka.clusters.0.serde.0.name: ProtobufFile
kafka.clusters.0.serde.0.className: com.provectus.kafka.ui.serdes.builtin.ProtobufRawSerde
kafka.clusters.0.serde.0.properties.protobufFilesDir: /protofiles/
kafka.clusters.0.serde.0.properties.protobufMessageName: io.cloudevents.v1.CloudEvent # default type for values
kafka.clusters.0.serde.0.properties.protobufMessageNameByTopic.ais-fused: io.cloudevents.v1.CloudEvent
volumes:
- "./proto/:/protofiles"

However I dont see Protobuf getting picked up by UI.

[Haarolean]

@supriya-albal-polestar let's keep the discussion on topic, feel free to raise a discussion or join us on discord for unrelated things.