[HOTFIX] applying PR #6109 to rc/2026-01-02 #6110
Conversation
## Description of changes _Summarize the changes made by this PR._ - Improvements & Bug fixes - N/A - New functionality - Eagerly commit blocks in ordered blockfile writer to avoid excessive memory usage ## Test plan _How are these changes tested?_ - [ ] Tests pass locally with `pytest` for python, `yarn test` for js, `cargo test` for rust ## Migration plan _Are there any migrations, or any forwards/backwards compatibility changes needed in order to make sure this change deploys reliably?_ ## Observability plan _What is the plan to instrument and monitor this change?_ ## Documentation Changes _Are all docstrings for user-facing APIs updated if required? Do we need to make documentation changes in the [docs section](https://github.com/chroma-core/chroma/tree/main/docs/docs.trychroma.com)?_
Reviewer ChecklistPlease leverage this checklist to ensure your code review is thorough before approving Testing, Bugs, Errors, Logs, Documentation
System Compatibility
Quality
|
|
Eager block commits in ordered blockfile writer Updates Key Changes• Replaced Affected Areas• rust/blockstore/src/arrow/ordered_blockfile_writer.rs This summary was automatically generated by @propel-code-bot |
![propel-code-bot[bot]](https://avatars.githubusercontent.com/in/1179004?s=60&v=4){kind=small}
| if delta.get_size::<K, V>() > self.root.max_block_size_bytes { | ||
| let split_blocks = delta.split::<K, V>(self.root.max_block_size_bytes); | ||
| for (split_key, split_delta) in split_blocks { | ||
| self.root | ||
| .sparse_index | ||
| .add_block(split_key, split_delta.id) | ||
| .map_err(|e| Box::new(e) as Box<dyn ChromaError>)?; | ||
| let block = self.block_manager.commit::<K, V>(split_delta).await; | ||
| inner.completed_blocks.push(block); | ||
| } | ||
| } | ||
| let block = self.block_manager.commit::<K, V>(delta).await; | ||
| inner.completed_blocks.push(block); |
There was a problem hiding this comment.
[Maintainability] This logic for splitting an oversized delta and committing its pieces is duplicated in the set method on lines 319-331.
To improve maintainability and adhere to the DRY (Don't Repeat Yourself) principle, consider extracting this logic into a private helper method. This new method could take the delta and inner state, perform the necessary splitting and committing, and then be called from both complete_current_delta and set.
Context for Agents
This logic for splitting an oversized delta and committing its pieces is duplicated in the `set` method on lines 319-331.
To improve maintainability and adhere to the DRY (Don't Repeat Yourself) principle, consider extracting this logic into a private helper method. This new method could take the `delta` and `inner` state, perform the necessary splitting and committing, and then be called from both `complete_current_delta` and `set`.
File: rust/blockstore/src/arrow/ordered_blockfile_writer.rs
Line: 211
This PR cherry-picks the commit acde656 onto rc/2026-01-02. If there are unresolved conflicts, please resolve them manually.