KAFKA-19019: Add support for remote storage fetch for share groups #19437
Conversation
github-actions
bot
added
triage
adixitconfluent
changed the title
There was a problem hiding this comment.
@adixitconfluent : Thanks for the PR. Made a pass of the non-testing file. Left a few comments.
There was a problem hiding this comment.
@adixitconfluent : Thanks for the updated PR. A few more comments.
There was a problem hiding this comment.
@adixitconfluent : Thanks for the updated PR. A few more comments.
There was a problem hiding this comment.
@adixitconfluent : Thanks for the updated PR. A few more comments.
There was a problem hiding this comment.
@adixitconfluent : Thanks for the updated PR. A couple of more comments.
There was a problem hiding this comment.
@adixitconfluent : Thanks for the updated PR. Just a minor comment.
| releasePartitionLocks(topicPartitionData.keySet()); | ||
| partitionsAcquired.clear(); | ||
| localPartitionsAlreadyFetched.clear(); | ||
| return forceComplete(); |
There was a problem hiding this comment.
Should we just call forceCompleteRequest() here to for consistency? We could optimize releasePartitionLocksAndAddToActionQueue() so that if doesn't add to the action queue if the acquired partition set is empty.
What
This PR adds the support for remote storage fetch for share groups.
Limitation
There is a limitation in remote storage fetch for consumer groups that
we can only perform remote fetch for a single topic partition in a fetch
request. Since, the logic of share fetch requests is largely based on
how consumer
groups work, we are following similar logic in implementing remote
storage fetch. However, this problem
should be addressed as part of KAFKA-19133 which should help us perform
fetch for multiple remote fetch topic partition in a single share fetch
request.
Testing
I have followed the AK
documentation
to test my code locally (by adopting
LocalTieredStorage.java) and withthe help of unit tests.