KAFKA-17411: Use shared cache for Task offset sums #17715
Conversation
|
@mjsax @cadonna @ableegoldman Part 2 of KIP-1035. This actually makes use of the "startup Tasks" by caching their offset sums on startup and using that for rebalances instead of the We still use the |
nicktelford
changed the title
Instead of reading Task state offsets for non-open Tasks from the `.checkpoint` file, we now maintain an in-memory cache of the latest changelog offsets for every Task on the instance. On start-up, this cache is seeded with the changelog offsets for every on-disk StateStore. Running Active and Standby Tasks then update this cache on every checkpoint to ensure it always reflects the offsets on-disk. This breaks the tight coupling between `TaskManager` and `.checkpoint` files, which will enable us to remove `.checkpoint` files in a later commit as part of KIP-1035.
nicktelford
force-pushed
the
KIP-1035-b
branch
from
f828c6e to
fc7eeb8
Compare
|
Rebased against |
|
This PR is being marked as stale since it has not had any activity in 90 days. If you If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact). If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed. |
|
hey @nicktelford sorry this PR was neglected for so long, is it still ready for review? might need to be rebased first? |
|
@ableegoldman Yeah, that's not a surprise. I should be able to get to it next week, or maybe the week after. |
|
SG! Ping me again when it's ready 🙂 |
|
This PR is being marked as stale since it has not had any activity in 90 days. If you If you are having difficulty finding a reviewer, please reach out on the [mailing list](https://kafka.apache.org/contact). If this PR is no longer valid or desired, please feel free to close it. If no activity occurs in the next 30 days, it will be automatically closed. |
Instead of reading Task state offsets for non-open Tasks from the
.checkpointfile, we now maintain an in-memory cache of the latest changelog offsets for every Task on the instance.On start-up, this cache is seeded with the changelog offsets for every on-disk StateStore. Running Active and Standby Tasks then update this cache on every checkpoint to ensure it always reflects the offsets on-disk.
This breaks the tight coupling between
TaskManagerand.checkpointfiles, which will enable us to remove.checkpointfiles in a later commit as part of KIP-1035.