Describe the feature
Support Kafka exactly-once semantics (EOS) in the Dapr Kafka pub/sub component so that applications can get at-most-once delivery per publish when enabled via component configuration. Users would be able to opt in (e.g. via metadata) and have the component ensure that each published message is delivered and committed in a way that avoids duplicate processing under normal operation.
Kafka has supported exactly-once semantics since 0.11 (transactional produce/consume and offset commit). (see e.g. Plans to support upcoming Kafka v0.11 ? (#901) and Exactly Once arrived with kafka 0.11 ? (#977)); the feature request is to expose and certify this capability in the Dapr component. At a high level, enabling EOS would mean:
- producing within a transaction (so sends are atomic),
- consuming with read-committed isolation
- committing consumer offsets in the same transaction so processing and offset commit succeed or fail together—avoiding duplicates and double-processing when the component is used as configured.
Business value: Applications that need exactly-once or no-duplicate processing over Kafka could rely on Dapr’s Kafka component instead of building their own transaction handling, which is especially useful for event-driven workflows, audit trails, and pipelines that must not double-process messages.
Release Note
RELEASE NOTE: ADD Kafka pub/sub component supports exactly-once semantics (EOS) via optional metadata, enabling transactional produce and consume so applications can avoid duplicate processing.
Describe the feature
Support Kafka exactly-once semantics (EOS) in the Dapr Kafka pub/sub component so that applications can get at-most-once delivery per publish when enabled via component configuration. Users would be able to opt in (e.g. via metadata) and have the component ensure that each published message is delivered and committed in a way that avoids duplicate processing under normal operation.
Kafka has supported exactly-once semantics since 0.11 (transactional produce/consume and offset commit). (see e.g. Plans to support upcoming Kafka v0.11 ? (#901) and Exactly Once arrived with kafka 0.11 ? (#977)); the feature request is to expose and certify this capability in the Dapr component. At a high level, enabling EOS would mean:
Business value: Applications that need exactly-once or no-duplicate processing over Kafka could rely on Dapr’s Kafka component instead of building their own transaction handling, which is especially useful for event-driven workflows, audit trails, and pipelines that must not double-process messages.
Release Note
RELEASE NOTE: ADD Kafka pub/sub component supports exactly-once semantics (EOS) via optional metadata, enabling transactional produce and consume so applications can avoid duplicate processing.