Skip to content

KAFKA-15630: Improve documentation of offset.lag.max #21357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
k0b3rIT wants to merge 1 commit into
base: trunk
Choose a base branch
from
+11 −1
Open

KAFKA-15630: Improve documentation of offset.lag.max #21357

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion connect/mirror/src/main/java/org/apache/kafka/connect/mirror/MirrorSourceConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,17 @@ public class MirrorSourceConfig extends MirrorConnectorConfig {
public static final Class<?> CONFIG_PROPERTY_FILTER_CLASS_DEFAULT = DefaultConfigPropertyFilter.class;

public static final String OFFSET_LAG_MAX = "offset.lag.max";
private static final String OFFSET_LAG_MAX_DOC = "How out-of-sync a remote partition can be before it is resynced.";
private static final String OFFSET_LAG_MAX_DOC = "Defines the maximum lag (downstream offset - last synced downstream offset) threshold before " +
"MirrorSourceTask generates an offset-sync (upstream-downstream offset mapping) record during partition replication. " +
"In practice: Setting this value to 100 will produce an offset-sync at least after every 100 records. " +
"Lowering this value increases the precision of the consumer offset checkpoints created by the MirrorCheckpointTask based on the offset-syncs. " +
"This is particularly beneficial for low-lag consumers reading from the end of a topic. Conversely, when a consumer operates with high lag, " +
"checkpoint precision may degrade due to internal optimizations. " +
"Setting this value to 0 forces an offset-sync for every replicated record. While this maximizes precision, " +
"it may incur performance and storage overhead for high-throughput topics. " +
"A higher value can reduce the footprint of the offset-sync topic, " +
"but may result in excessive record duplication (consumer reprocessing) during a failover. " +
"Partition Count * offset.lag.max = Approximate duplicated record count (Actual value can be lower or even higher depending on timing and consumer lag)";
public static final long OFFSET_LAG_MAX_DEFAULT = 100L;

public static final String HEARTBEATS_REPLICATION_ENABLED = "heartbeats.replication" + ENABLED_SUFFIX;
Expand Down