Skip to content

[v25.3.x] Fix case where messages can be written multiple times in Iceberg #29397

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
vbotbuildovich wants to merge 3 commits into
base: v25.3.x
Choose a base branch
from
Open

[v25.3.x] Fix case where messages can be written multiple times in Iceberg #29397

vbotbuildovich wants to merge 3 commits into from

Conversation

@vbotbuildovich
Copy link
Collaborator

@vbotbuildovich vbotbuildovich commented Jan 26, 2026

Backport of PR #29251

@ballard26 @vbotbuildovich
datalake: allow oom injection in noop_mem_tracker
9fe5c9d 
(cherry picked from commit c7c8c5c)
@ballard26 @vbotbuildovich
datalake: always return writer_error::ok on successful row writes
ad959a0 
Callers of `serde_parquet_writer::add_data_struct` assume that the data
struct wasn't successfully written to the file if `writer_error::ok`
isn't returned. However, it's possible for `_writer.write_row(...)` to
be successful then have some disk/memory reservation error. Prior to
this commit these errors would result in the message being written
twice as the callers would assume that the first write had failed.

(cherry picked from commit efc3a22)
@ballard26 @vbotbuildovich
datalake: fix TestCleanupAfterOOMError unit test
72a639d 
In a previous commit `serde_parquet_writer` was changed to always return
an `ok` result when the record is written. This resulted in the flush at
the translation_task level to no longer return a no data error. The unit
test is therefore modified in this commit to reflect the new behavior.

(cherry picked from commit 7f506e3)
@vbotbuildovich vbotbuildovich added this to the v25.3.x-next milestone Jan 26, 2026
@vbotbuildovich vbotbuildovich added the kind/backport PRs targeting a stable branch label Jan 26, 2026
@vbotbuildovich
Copy link
Collaborator Author

vbotbuildovich commented Jan 26, 2026

CI test results

test results on build#79627
test_class test_method test_arguments test_kind job_url test_status passed reason test_history
ConsumerGroupBalancingTest test_coordinator_nodes_balance null integration https://buildkite.com/redpanda/redpanda/builds/79627#019bfb1b-1507-42ed-9253-ddc392223e68 FLAKY 10/11 Test PASSES after retries.No significant increase in flaky rate(baseline=0.0000, p0=1.0000, reject_threshold=0.0100. adj_baseline=0.1000, p1=0.3487, trust_threshold=0.5000) https://redpanda.metabaseapp.com/dashboard/87-tests?tab=142-dt-individual-test-history&test_class=ConsumerGroupBalancingTest&test_method=test_coordinator_nodes_balance

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ballard26 ballard26 Awaiting requested review from ballard26

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

area/redpanda kind/backport PRs targeting a stable branch

Projects

None yet

Milestone

v25.3.x-next

Development

Successfully merging this pull request may close these issues.

2 participants

@vbotbuildovich @ballard26