Add bytes-based flushing to source collectors - #1102
Open
46bit wants to merge 1 commit into
Open
Conversation
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
46bit
force-pushed
the
miki/bytes-based-collector-flushing
branch
2 times, most recently
from
July 10, 2026 16:45
9ebfc42 to
2ccfaf7
Compare
mwylde
approved these changes
Jul 23, 2026
46bit
force-pushed
the
miki/bytes-based-collector-flushing
branch
from
July 23, 2026 11:29
2ccfaf7 to
0ff557e
Compare
Sources read data in and collect it up into batches. Batches are flushed based on configurable criteria: until now this has been the number of records collected and how long they have been waiting. This commit adds another criteria for flushing, based on the memory usage of the records. `source-batch-max-bytes`, disabled by default, allows triggering flushing when a certain amount of data has been collected. This new feature enables two useful behaviours: 1) Large records: reduce the risk of OOMs when memory-constrained 2) Small records: create larger, more-efficient batches Memory usage is estimated based on input bytes, rather than by instrumenting Arrow buffer memory. Ideally we would implement this based on Arrow buffer usage. However, Arrow's `ArrayBuilder` trait doesn't expose memory size and we would need to maintain a lot of code ourselves to make it work. I think it's reasonable to use input bytes as an approximation.
mwylde
force-pushed
the
miki/bytes-based-collector-flushing
branch
from
July 27, 2026 19:21
0ff557e to
32abbf2
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Sources read data in and collect it up into batches. Batches are flushed based on configurable criteria: until now this has been the number of records collected and how long they have been waiting.
This commit adds another criteria for flushing, based on the memory usage of the records.
source-batch-max-bytes, disabled by default, allows triggering flushing when a certain amount of data has been collected.Why
This new feature enables two useful behaviours:
How
Memory usage is estimated based on input bytes, rather than by instrumenting Arrow buffer memory.
Ideally we would implement this based on Arrow buffer usage. However, Arrow's
ArrayBuildertrait doesn't expose memory size and we would need to maintain a lot of code ourselves to make it work. I think it's reasonable to use input bytes as an approximation.