Skip to content

fix: Split batches correctly by InsertOp#80

Merged
peasee merged 2 commits into
trunkfrom
peasee/260218-split-batches-by-insertop
Feb 19, 2026
Merged

fix: Split batches correctly by InsertOp#80
peasee merged 2 commits into
trunkfrom
peasee/260218-split-batches-by-insertop

Conversation

@peasee
Copy link
Copy Markdown
Contributor

@peasee peasee commented Feb 19, 2026

🗣 Description

  • Ensures that batches read from S3Storage do not have an InsertOp assigned to the entire file - as a single file could contain multiple different insert ops
  • Updates ETL to split the batches by their largest contiguous ranges of the same InsertOp, and applies these with write operations to the configured sink

@peasee peasee self-assigned this Feb 19, 2026
@peasee peasee added the enhancement New feature or request label Feb 19, 2026
@peasee peasee enabled auto-merge (squash) February 19, 2026 02:35
@peasee peasee merged commit dab392c into trunk Feb 19, 2026
9 checks passed
@peasee peasee deleted the peasee/260218-split-batches-by-insertop branch February 19, 2026 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants