Table restore error when the server has compacted the changelog-topic by segment time #176
Description
Hello
When I restart my application after a few time I get this error: [^---Recovery]: No event received for active tp TopicPartition(topic='philipp.match.04-matchtable-changelog', partition=0) since recovery start (started 33.43 seconds ago) on Recoverying for the Table.
Full traceback
[2021-08-05 14:11:10,553] [6] [INFO] [^---Recovery]: Highwater for active changelog partitions:
┌Highwater - Active─────────────────────┬───────────┬───────────┐
│ topic │ partition │ highwater │
├───────────────────────────────────────┼───────────┼───────────┤
│ philipp.match.04-matchtable-changelog │ 0 │ 19477 │
└───────────────────────────────────────┴───────────┴───────────┘
[2021-08-05 14:11:10,589] [6] [INFO] [^---Recovery]: active offsets at start of reading:
┌Reading Starts At - Active─────────────┬───────────┬────────┐
│ topic │ partition │ offset │
├───────────────────────────────────────┼───────────┼────────┤
│ philipp.match.04-matchtable-changelog │ 0 │ 17091 │
└───────────────────────────────────────┴───────────┴────────┘
[2021-08-05 14:11:10,596] [6] [INFO] [^---Recovery]: Restoring state from changelog topics...
[2021-08-05 14:11:10,597] [6] [INFO] [^---Recovery]: Resuming flow...
[2021-08-05 14:11:10,600] [6] [INFO] [^---Fetcher]: Starting...
[2021-08-05 14:11:12,132] [6] [INFO] Fetch offset 17091 is out of range for partition TopicPartition(topic='philipp.match.04-matchtable-changelog', partition=0), resetting offset
[2021-08-05 14:11:44,025] [6] [WARNING] [^---Recovery]: No event received for active tp TopicPartition(topic='philipp.match.04-matchtable-changelog', partition=0) since recovery start (started 33.43 seconds ago)
[2021-08-05 14:11:49,028] [6] [WARNING] [^---Recovery]: No event received for active tp TopicPartition(topic='philipp.match.04-matchtable-changelog', partition=0) since recovery start (started 38.43 seconds ago)
[2021-08-05 14:11:54,030] [6] [WARNING] [^---Recovery]: No event received for active tp TopicPartition(topic='philipp.match.04-matchtable-changelog', partition=0) since recovery start (started 43.43 seconds ago)
[2021-08-05 14:11:59,031] [6] [WARNING] [^---Recovery]: No event received for active tp TopicPartition(topic='philipp.match.04-matchtable-changelog', partition=0) since recovery start (started 48.44 seconds ago) Steps to reproduce
I have set the segmenttime to 10 minutes, the retentiontime to 100 minuntes and the cleanup policy to compact,delete. I use Confluent Cloud as Kafka Service.
extra_topic_configs = {
'segment.ms': 600000, # 10 minutes
'retention.ms': 6000000, # 100 minutes
'cleanup.policy': 'compact,delete'}The error occurs when I switch off the app for about 30 minutes and then restart it.
When I wait much more than the retention time (for example 10 hours), the the error does not occur and the App starts with an empty Table. Just as I would expect.
In my opinion, the error comes from the fact that some messages are no longer available due to the segment time. The Kafka Service has already compacted and deleted several older messages with the same key. As a result, the offset 17091 is no longer available, but the changelog topic is not empty either, otherwise it would start with an empty table.
Expected behavior
Faust starts with the next available offset from the changelog-topic for Recovery
Actual behavior
If the changelog-topic is not empty, a warning is issued in an endless loop and the agent does not working.
Versions
- Python version 3.7.9
- Faust version 0.6.9
Has anyone of you had the same problem or a solution for me?