Skip to content

Commit c40f0d9

Browse files
parasebadcherian
andauthored
Document Session.fork (#1216)
* Document Session.fork * Update icechunk-python/python/icechunk/session.py Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com> --------- Co-authored-by: Deepak Cherian <dcherian@users.noreply.github.com>
1 parent 1b96387 commit c40f0d9

File tree

1 file changed

+18
-0
lines changed

1 file changed

+18
-0
lines changed

icechunk-python/python/icechunk/session.py

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -359,6 +359,24 @@ async def rebase_async(self, solver: ConflictSolver) -> None:
359359
await self._session.rebase_async(solver)
360360

361361
def fork(self) -> "ForkSession":
362+
"""
363+
Create a child session that can be pickled to a worker job and later merged.
364+
365+
This method supports Icechunk's distributed, collaborative jobs. A coordinator task creates a new session using
366+
`Repository.writable_session`. Then `Session.fork` is called repeatedly to create as many serializable sessions
367+
as worker jobs. Each new `ForkSession` is pickled to the worker that uses it to do all its writes.
368+
Finally, the `ForkSessions` are pickled back to the coordinator that uses `ForkSession.merge` to merge them
369+
back into the original session and `commit`.
370+
371+
Learn more about collaborative writes at https://icechunk.io/en/latest/parallel/
372+
373+
Raises
374+
------
375+
ValueError
376+
When `self` already has uncommitted changes.
377+
ValueError
378+
When `self` is read-only.
379+
"""
362380
if self.has_uncommitted_changes:
363381
raise ValueError(
364382
"Cannot fork a Session with uncommitted changes. "

0 commit comments

Comments
 (0)