Skip to content

Conversation

@iczero
Copy link
Contributor

@iczero iczero commented Dec 15, 2025

  • Introduce CompactionFilter trait which can elect to drop a kv or modify its value
  • Introduce ItemAccessor for CompactionFilter which can access key/value within the compaction filter
  • Rename ExpiredKvCallback to DroppedKvCallback since it's not really exclusively for "expired" anymore
  • Add filter support to CompactionStream

Compaction filters replace dropped KVs with a tombstone. If we are compacting to the last level, the tombstone will subsequently be removed by the CompactionStream. If the value is changed, it is replaced in place or written out to a new blob file as necessary.

Closes #159

@iczero iczero force-pushed the compaction-filters-2 branch from 558deaa to d86e02f Compare December 15, 2025 10:59
@codecov
Copy link

codecov bot commented Dec 15, 2025

Codecov Report

❌ Patch coverage is 97.76536% with 4 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
src/compaction/filter.rs 95.87% 4 Missing ⚠️

📢 Thoughts on this report? Let us know!

@iczero iczero marked this pull request as ready for review December 20, 2025 10:31
@iczero iczero force-pushed the compaction-filters-2 branch from 35bc93e to d7467c3 Compare December 21, 2025 14:18
@iczero iczero force-pushed the compaction-filters-2 branch from a388791 to 11e7363 Compare December 22, 2025 06:07
@iczero iczero requested a review from marvin-j97 December 22, 2025 06:41
@iczero iczero requested a review from marvin-j97 December 23, 2025 04:54
@iczero
Copy link
Contributor Author

iczero commented Dec 23, 2025

I've added a finish method to CompactionFilter as well. It's possible to accomplish much of the same using Drop, but finish takes ownership of self and as such is a tad bit more flexible.

@marvin-j97
Copy link
Contributor

I've been thinking of calling this "compaction mappers" instead of "compaction filters" because "filters" is already quite a loaded term (AMQ filters and filter() in Rust), while this operation is more akin to a filter_map really.

@iczero
Copy link
Contributor Author

iczero commented Jan 4, 2026

In my opinion, "filter" would probably fit better in this case, but I'm fine with either name. Should I change it?

@marvin-j97
Copy link
Contributor

In my opinion, "filter" would probably fit better in this case, but I'm fine with either name. Should I change it?

I think it's fine either way

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Compaction filters

2 participants