Skip to content

feat(v2): metadata index retention policy #4148

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kolesnikovae
Copy link
Collaborator

@kolesnikovae kolesnikovae commented Apr 28, 2025

The change adds a new component that is responsible for enforcement of the retention policy.

It's implemented as follows:

  1. On the leader node, we periodically check which partition shards have passed the retention period threshold. The state is accessed via the Leader Read interface: it always reads the local state.
  2. The component creates tombstones for the partition shards (not the blocks they contain) and issues a new raft command – truncate index.
  3. The command handler removes specified partition shards from the index.
  4. Tombstones then follow the standard tombstones's life-cycle: eventually, they are included into a compaction job.
  5. Once the compaction job is added to the scheduler, we remove the associated tombstone object.
  6. A compaction worker deletes all the objects satisfying the prefix that corresponds the partition tombstone (it includes the partition timestamp) – this part is not implemented yet.

@kolesnikovae kolesnikovae changed the title add basic implementation feat(v2): metadata index retention policy Apr 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant