feat: flush hook extension point#8145
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a FlushHook extension point for the mito2 engine, allowing plugins to receive notifications when SST files are written or the manifest is updated during flush and compaction operations. The implementation includes a new PkCollectingIter to gather unique primary keys during these processes. Feedback from the review highlights a critical bug in the primary key column indexing logic and a serialization issue where hook-related data is skipped in MergeOutput, which would break remote compaction. Additionally, there are suggestions to optimize performance by using references instead of cloning FileMeta and a warning regarding potential memory exhaustion when collecting a large volume of primary keys.
9a15670 to
9ac92e5
Compare
I hereby agree to the terms of the GreptimeDB CLA.
Refer to a related PR or issue link (optional)
What's changed and what's your intention?
This patch adds extension points and plugin resgistration for flush/compaction.
PR Checklist
Please convert it to a draft if some of the following conditions are not met.