Kafka source#966

Open

MrAnno wants to merge 32 commits into

axoflow:mainfrom

MrAnno:kafka-source

Commits on Feb 25, 2026

!!! mainloop/threaded-source: FIXME

HofiOne
authored and
MrAnno
committed
msg-format: add msg_format_from_string()
MrAnno
committed
kafka: make kafka() and kafka-c() aliases

HofiOne
authored and
MrAnno
committed
kafka-dest: add missing headers to the cmake file

HofiOne
authored and
MrAnno
committed
kafka-dest: moved declarations of reusable parts to kafka-internal.h

HofiOne
authored and
MrAnno
committed
kafka-dest: add missing raw message size stats update

HofiOne
authored and
MrAnno
committed
kafka-dest: fix include guard
MrAnno
committed
kafka: move none public/common parts to kafka-internal.c

HofiOne
authored and
MrAnno
committed
kafka: add common base options, add separate optiosn to kafka dest

HofiOne
authored and
MrAnno
committed
kafka: add kafka_logging() option

HofiOne
authored and
MrAnno
committed
kafka-dest: refactor

HofiOne
authored and
MrAnno
committed
kafka-props: added kafka_property_list_find_not_empty func

HofiOne
authored and
MrAnno
committed
kafka: add KafkaOpaque as separate class which let us:

HofiOne
authored and
MrAnno
committed
kafka-internal: add kafka_msg_debug

HofiOne
authored and
MrAnno
committed
kafka-internal: added kafka_register_counters/kafka_unregister_counters

HofiOne
authored and
MrAnno
committed
kafka-internal: add source fields
MrAnno
committed
kafka-source: implement source

HofiOne
authored and
MrAnno
committed
kafka-source-persist: added persist state handling for both linear and none-linear consuming

HofiOne
authored and
MrAnno
committed
kafka-source: added disable_bookmarks option, do_not_use_bookmark renamed to ignore_saved_bookmarks and synced its handling in the persist state maintenance

HofiOne
authored and
MrAnno
committed
kafka-source: workaround of the assign method issue (probably librdkfka bug) at startup without a connection

HofiOne
authored and
MrAnno
committed
kafka-source-persist: Separate better the allocation and the init part, store owner references

HofiOne
authored and
MrAnno
committed
kafka-source: add persist store option and remote storing impl

HofiOne
authored and
MrAnno
committed
kafka-source: add store_metadata option

HofiOne
authored and
MrAnno
committed
kafka-source: added driver tag to logs where they were missing

HofiOne
authored and
MrAnno
committed
kafka-source: fix persist ready funcs, rename ref-unref conforming to the already exiting nameing convention

HofiOne
authored and
MrAnno
committed
kafka-source: do not try to lock the perists multiple places, handle the rebalance event correctly, synced instead

HofiOne
authored and
MrAnno
committed
kafka-source: do not try to reassign the partitions manually without fully reinitiation of the consumer (will not always work)

HofiOne
authored and
MrAnno
committed
kafka-source: let the locally stored bookmarks be committed even if the persist is invalidated, as those are not affected by the remote connection/partition assignment state

HofiOne
authored and
MrAnno
committed
style: astyle formatting issues fixed

HofiOne
authored and
MrAnno
committed
kafka-source: First round adjustments according to the review comments

HofiOne
authored and
MrAnno
committed
kafka-source: second round adjustments according to the review comments

HofiOne
authored and
MrAnno
committed
kafka: upgrade license from GPL-2.0-or-later to GPL-3.0-or-later
MrAnno
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka source#966

Kafka source#966
MrAnno wants to merge 32 commits into
axoflow:mainfrom
MrAnno:kafka-source

Commits on Feb 25, 2026

!!! mainloop/threaded-source: FIXME

msg-format: add msg_format_from_string()

kafka: make kafka() and kafka-c() aliases

kafka-dest: add missing headers to the cmake file

kafka-dest: moved declarations of reusable parts to kafka-internal.h

kafka-dest: add missing raw message size stats update

kafka-dest: fix include guard

kafka: move none public/common parts to kafka-internal.c

kafka: add common base options, add separate optiosn to kafka dest

kafka: add kafka_logging() option

kafka-dest: refactor

kafka-props: added kafka_property_list_find_not_empty func

kafka: add KafkaOpaque as separate class which let us:

kafka-internal: add kafka_msg_debug

kafka-internal: added kafka_register_counters/kafka_unregister_counters

kafka-internal: add source fields

kafka-source: implement source

kafka-source-persist: added persist state handling for both linear and none-linear consuming

kafka-source: added disable_bookmarks option, do_not_use_bookmark renamed to ignore_saved_bookmarks and synced its handling in the persist state maintenance

kafka-source: workaround of the assign method issue (probably librdkfka bug) at startup without a connection

kafka-source-persist: Separate better the allocation and the init part, store owner references

kafka-source: add persist store option and remote storing impl

kafka-source: add store_metadata option

kafka-source: added driver tag to logs where they were missing

kafka-source: fix persist ready funcs, rename ref-unref conforming to the already exiting nameing convention

kafka-source: do not try to lock the perists multiple places, handle the rebalance event correctly, synced instead

kafka-source: do not try to reassign the partitions manually without fully reinitiation of the consumer (will not always work)

kafka-source: let the locally stored bookmarks be committed even if the persist is invalidated, as those are not affected by the remote connection/partition assignment state

style: astyle formatting issues fixed

kafka-source: First round adjustments according to the review comments

kafka-source: second round adjustments according to the review comments

kafka: upgrade license from GPL-2.0-or-later to GPL-3.0-or-later