Default consumer offset reset policy - earliest

[srolija]

By default Kafka Consumer has auto.offset.reset policy configured to latest. But it looks that the implementation in this connector is reverse -- unless it is configured it will start with earliest.

mirus/src/main/java/com/salesforce/mirus/MirusSourceTask.java

Lines 161 to 167 in 4b48244

	String offsetReset = (String) consumerProperties.get("auto.offset.reset");
	if ("latest".equalsIgnoreCase(offsetReset)) {
	logger.trace("Seeking to end");
	consumer.seekToEnd(Collections.singletonList(tp));
	} else {
	logger.trace("Seeking to beginning");
	consumer.seekToBeginning(Collections.singletonList(tp));

And that value is extracted from the consumer prefixed ones:

mirus/src/main/java/com/salesforce/mirus/config/SourceConfig.java

Line 51 in 4b48244

return simpleConfig.originalsWithPrefix("consumer.");

Would it make sense to make the default the same as the normal connector to keep it consistent with normal consumer groups?

[pdavidson100]

Thanks for your suggestion @srolija! The Mirus Source Connector is primarily intended for high-reliability data replication use cases, so our defaults are selected to minimize the risk of data loss at all costs — even at expense of increased risk of duplicate data. By using the earliest policy we guarantee that all available data is replicated, even in exceptional circumstances. This policy covers us when new topics are added to the topic regex and, importantly, also in rare instances where a Kafka bug causes the current offset to become invalid and the consumer offsets are reinitialized. This is something we have occasionally seen, particularly in earlier Kafka releases and using latest in that situation would certainly lead to data loss.

[srolija]

Completely get it.

Would it then make sense just to note the changes from the default consumer group? Asking since we had an issue where it started replicating enormous topic; and based on the docs we didn't understand that it has non-default configuration.

[pdavidson100]

Yes, that does make sense - we should update the the docs.