Skip to content

Commit 05ee25f

Browse files
authored
[Data] Propagate branch in _append_and_commit(..) for iceberg overwrite save mode (#63922)
Fix bug where if a user passes `branch` as an `overwrite_kwarg` the delete lands on staging but the subsequent append lands on main (the default in `_append_and_commit`) Now we should thread through this arg to the append phase as well. Signed-off-by: Ayush Kumar <ayushk7102@gmail.com>
1 parent 8f6dd41 commit 05ee25f

1 file changed

Lines changed: 3 additions & 2 deletions

File tree

python/ray/data/_internal/datasource/iceberg_datasink.py

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -841,8 +841,9 @@ def _commit_overwrite(
841841
**self._overwrite_kwargs,
842842
)
843843

844-
# Append new data files and commit
845-
self._append_and_commit(txn, data_files)
844+
# Append on the same branch the delete targeted (defaults to "main").
845+
branch = self._overwrite_kwargs.get("branch", "main")
846+
self._append_and_commit(txn, data_files, branch=branch)
846847

847848
def on_write_complete(self, write_result: WriteResult) -> None:
848849
"""

0 commit comments

Comments
 (0)