-
Notifications
You must be signed in to change notification settings - Fork 99
Open
Description
I'm trying to set my Producer to use Zstd compression, like this:
// Prepare the configuration
kafka::Properties props({
{"bootstrap.servers", {m_BrokerList}} // The Kafka Brokers
});
// The default partitioner is crc_random, but Java decided to use murmur2_random partitioner instead so
// we need to do the same to be Java compatible
props.put("partitioner", "murmur2_random");
// Use Zstd compression for these messages
props.put("compression.type", "zstd");
// Intercept logging, so that it logs using our DataDogLogger and not directly to stdout in it's own format
props.put("log_level", "4"); // Warning level
props.put("log_cb", KafkaLog); // Set the logging callback
// Create a producer instance
m_KafkaProducer = std::make_shared<kafka::clients::producer::KafkaProducer>(props);
However, this results in the following error in my logs:
Error from Kafka Lib (Level3): KafkaProducer[18f0a03a-ce6be2ba] failed to be initialized with property[compression.type:zstd], result[-1]: Unsupported value "zstd" for configuration property "compression.codec": libzstd not available at build time
It looks like the version of librdkafka used by Modern-cpp-kafka wasn't built with the Zstd library.
Is it possible to enable Zstd compression with Modern-Cpp-Kafka?
I'm guessing I'd need to build librdkafka with Zstd support and use that instead of the version included by Modern-Cpp-Kafka itself?
Thanks
Ben
Metadata
Metadata
Assignees
Labels
No labels