statement-store: reduce index memory footprint with about 200 bytes #10894
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Alexandru Gheorghe <[email protected]>
AndreiEres
reviewed
Jan 23, 2026
| if let Some((topics, key)) = self.topics_and_keys.remove(hash) { | ||
| for t in topics.into_iter().flatten() { | ||
| // Read statement from database to get topics and decryption key | ||
| let statement = db |
Contributor
There was a problem hiding this comment.
I don't like the idea of operating with the DB inside index?
Contributor
Author
There was a problem hiding this comment.
What is your concern here ?
Signed-off-by: Alexandru Gheorghe <[email protected]>
bkchr
reviewed
Jan 28, 2026
Member
bkchr
left a comment
bkchr
left a comment
There was a problem hiding this comment.
So you are trading memory footprint for IO? :D
| // Fallback: iterate through all indexes if database read failed | ||
| log::warn!( | ||
| target: LOG_TARGET, | ||
| "Failed to read statement {:?} from database, falling back to full index scan", |
Member
There was a problem hiding this comment.
Where is this reading from database? Also can we make this debug?
Member
There was a problem hiding this comment.
I see where it is reading from db :D
| } | ||
| } else { | ||
| // Fallback: iterate through all indexes if database read failed | ||
| log::warn!( |
Member
There was a problem hiding this comment.
Suggested change
| log::warn!( | |
| log::debug!( |
Contributor
Author
Yes, that's my intention with this PR, with the caveat that expiring statements is not something we do very often and it is not time sensitive. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
topics_and_keysis eating about 200 bytes per statement, which is about 30% of the contribution for each statement.Since this structure was need only when we clean up of statement it makes more sense to just read the statement from the database instead of paying such a high memory price for each statement.