ConsumerGroup resetting offset when topic leader changed

[hardy-luo]

I encountered a suspected bug:
when a topic is migrated from one broker to another, the consumer will detect an out of range error, and then the offset will be reset to the latest according to the auto.offset.reset:latest, resulting in a large number of messages not being actually consumed.

Here are the steps to reproduce the test environment:

1. Create a 1 replica topic and write 100,000 messages. Most of the data is uploaded to S3, and a small amount is on disk (offset:0 has been uploaded to S3)

2. Initialize a group to consume from 0, and there is a large Lag at this time is 100,000.

3. Migrate the topic from broker0 to broker1

4. Start consumer, return a error message "Fetch position FetchPosition{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[xxx:9092 (id: 1 rack: apse1-az1)], epoch=29}} is out of range for partition topic-0, resetting offset" is displayed. At this time, the Lag becomes 0

My current solution is to set auto.offset.reset:none, but this does not seem to be a fundamental solution

kafka version:3.6.1

[jeqo]

Thanks @hardy-luo! This is happening on the broker-side, so most probably not solvable within the plugin itself.

3.6.1 is also a very early version with Tiered Storage support; could you try to reproduce this on 3.9.0 or 3.8.1?