switch to roll and shift - remove delete vacated

Author: ianhi

icechunk-python/docs/docs/moving-chunks.md

@@ -15,41 +15,27 @@ This enables **rolling time windows**—continuously updating datasets like fore
 
 | Method | Best For | Flexibility |
 |--------|----------|-------------|
-| [`shift_array`][icechunk.Session.shift_array] | Uniform shifts with edge handling | Simple—just specify offset and mode |
+| [`shift_array`][icechunk.Session.shift_array] | Shift with discard — rolling time windows | Simple—just specify offset |
+| [`roll_array`][icechunk.Session.roll_array] | Circular shift — no data loss | Simple—just specify offset |
 | [`reindex_array`][icechunk.Session.reindex_array] | Custom transformations | Maximum—you control every chunk |
 
 ## Offsets Are in Chunks, Not Elements
 
-Both methods work with **chunk indices**, not array indices. If your array has `chunk_size=2`, then an offset of `(-1,)` shifts by 1 chunk, which is 2 elements:
+All three methods work with **chunk indices**, not array indices. If your array has `chunk_size=2`, then an offset of `(-1,)` shifts by 1 chunk, which is 2 elements:
 
 ```python
 # With chunk_size=2:
-shift_array("/arr", (-1,), "wrap") #  → shifts by 1 chunk = 2 elements
-shift_array("/arr", (-2,), "wrap") # → shifts by 2 chunks = 4 elements
+shift_array("/arr", (-1,))  # → shifts by 1 chunk = 2 elements
+shift_array("/arr", (-2,))  # → shifts by 2 chunks = 4 elements
 ```
 
 Why chunks instead of elements? Because these are **metadata-only operations**. Shifting by partial chunks would require splitting and rewriting chunk data.
 
-For convenience, `shift_array` returns the shift converted to element space—so you don't need to manually track chunk sizes when determining where to write new data.
+For convenience, `shift_array` and `roll_array` return the shift converted to element space—so you don't need to manually track chunk sizes when determining where to write new data.
 
 ## shift_array { #shift_array }
 
-The [`shift_array`][icechunk.Session.shift_array] method moves all chunks by a fixed offset per dimension (negative to shift toward index 0, positive toward higher indices), with built-in handling for what happens at the boundaries. For convenience, it returns the **index shift** (`chunk_offset × chunk_size` for each dimension).
-
-### Shift Modes
-
-The `mode` parameter controls what happens to chunks that shift out of bounds:
-
-| Mode | Behavior | Data Loss |
-|------|----------|-----------|
-| `"wrap"` | Chunks wrap to the other side | None |
-| `"discard"` | Out-of-bounds chunks are dropped | Yes |
-
-You can use strings (`"wrap"`, `"discard"`) or the enum ([`ic.ShiftMode.WRAP`][icechunk.ShiftMode], [`ic.ShiftMode.DISCARD`][icechunk.ShiftMode]).
-
-#### WRAP Mode
-
-Chunks that shift out of one end reappear at the other—no data is lost.
+The [`shift_array`][icechunk.Session.shift_array] method moves all chunks by a fixed offset per dimension (negative to shift toward index 0, positive toward higher indices). Chunks that shift out of bounds are discarded, and vacated positions retain stale data — the caller typically writes new data there. It returns the **index shift** (`chunk_offset × chunk_size` for each dimension).
 
 ```python exec="on" session="chunks" source="material-block" result="code"
 import numpy as np
@@ -71,15 +57,15 @@ arr = zarr.create(
 arr[:] = np.arange(10)
 print("Before:", arr[:])
 
-session.shift_array("/arr", (-2,), "wrap")  # Shift left by 2 chunks
+session.shift_array("/arr", (-2,))  # Shift left by 2 chunks
 print("After: ", arr[:])
 ```
 
-Notice how `[0, 1, 2, 3]` wrapped around to the end.
+The chunks containing `[0, 1, 2, 3]` were discarded, and the last 4 positions retain stale data.
 
-#### DISCARD Mode
+### Preserving Data with Resize
 
-Out-of-bounds chunks are dropped, and vacated positions return the fill value.
+Chunks that shift out of bounds are lost. To preserve everything when shifting, resize first:
 
 ```python exec="on" session="chunks" source="material-block" result="code"
 repo = ic.Repository.create(ic.in_memory_storage())
@@ -95,35 +81,30 @@ arr = zarr.create(
 arr[:] = np.arange(10)
 print("Before:", arr[:])
 
-session.shift_array("/arr", (-2,), "discard")  # Shift left, discard overflow
+arr.resize((14,))  # Add space for 2 more chunks
+session.shift_array("/arr", (2,))
 print("After: ", arr[:])
 ```
 
-The chunks containing `[0, 1, 2, 3]` were discarded, and the vacated end filled with `-1`.
+### Example: Rolling Time Window
 
-### Preserving Data with Resize
+Imagine a sensor array storing the last 7 days of hourly readings—shape `(168,)` with one chunk per day `(24,)`. Each day, you want to discard the oldest day and make room for new data:
 
-With `"discard"` mode, chunks that shift out of bounds are lost. To preserve everything when shifting, resize first:
+```python
+# Each day: shift left by 1 chunk, discarding the oldest
+element_shift = session.shift_array("/sensors/temperature", (-1,))
+# element_shift = (-24,) — the shift in element space
 
-```python exec="on" session="chunks" source="material-block" result="code"
-repo = ic.Repository.create(ic.in_memory_storage())
-session = repo.writable_session("main")
-arr = zarr.create(
-    store=session.store,
-    path="arr",
-    shape=(10,),
-    chunks=(2,),
-    dtype="i4",
-    fill_value=-1,
-)
-arr[:] = np.arange(10)
-print("Before:", arr[:])
+# Write new day's data to the vacated region
+arr[element_shift[0]:] = todays_readings
 
-arr.resize((14,))  # Add space for 2 more chunks
-session.shift_array("/arr", (2,), "discard")
-print("After: ", arr[:])
+session.commit(f"Updated sensor data for {today}")
 ```
 
+The return value tells you exactly where to write new data—no need to manually track chunk sizes.
+
+This pattern works identically whether your array is 1 KB or 1 PB, and whether it's on local disk or cloud object storage—the shift is always instant with zero data transfer.
+
 ### Multi-dimensional Arrays
 
 For N-dimensional arrays, provide an offset for each dimension:
@@ -143,38 +124,14 @@ arr[:] = np.arange(24).reshape(6, 4)
 print("Original 6x4 array:")
 print(arr[:])
 
-session.shift_array("/arr2d", (1, 0), "discard")  # Shift down 1 chunk
+session.shift_array("/arr2d", (1, 0))  # Shift down 1 chunk
 print("\nAfter shift (1, 0):")
 print(arr[:])
 ```
 
-### Example: Rolling Time Window
+## roll_array { #roll_array }
 
-Imagine a sensor array storing the last 7 days of hourly readings—shape `(168,)` with one chunk per day `(24,)`. Each day, you want to discard the oldest day and make room for new data:
-
-```python
-# Each day: shift left by 1 chunk, discarding the oldest
-element_shift = session.shift_array("/sensors/temperature", (-1,), "discard")
-# element_shift = (-24,) — the shift in element space
-
-# Write new day's data to the vacated region
-arr[element_shift[0]:] = todays_readings
-
-session.commit(f"Updated sensor data for {today}")
-```
-
-The return value tells you exactly where to write new data—no need to manually track chunk sizes.
-
-This pattern works identically whether your array is 1 KB or 1 PB, and whether it's on local disk or cloud object storage—the shift is always instant with zero data transfer.
-
-## reindex_array { #reindex_array }
-
-For transformations that [`shift_array`][icechunk.Session.shift_array] can't express, [`reindex_array`][icechunk.Session.reindex_array] gives you complete control. You provide a function that maps each chunk's old position to its new position.
-
-Your function receives a chunk index (as a list) and returns:
-
-- A new index (as a list) to move the chunk there
-- `None` to discard the chunk
+The [`roll_array`][icechunk.Session.roll_array] method performs a circular shift — chunks that go out of one end wrap around to the other side. No data is lost.
 
 ```python exec="on" session="chunks" source="material-block" result="code"
 repo = ic.Repository.create(ic.in_memory_storage())
@@ -190,23 +147,20 @@ arr = zarr.create(
 arr[:] = np.arange(10)
 print("Before:", arr[:])
 
-def shift_and_filter(idx):
-    """Shift left by 2, discard chunks that would go negative."""
-    new_idx = idx[0] - 2
-    return None if new_idx < 0 else [new_idx]
-
-session.reindex_array("/arr", shift_and_filter, delete_vacated=True)
+session.roll_array("/arr", (-2,))  # Roll left by 2 chunks
 print("After: ", arr[:])
 ```
 
-### The delete_vacated Parameter
+Notice how `[0, 1, 2, 3]` wrapped around to the end.
+
+## reindex_array { #reindex_array }
 
-The `delete_vacated` parameter controls what happens to source positions after chunks move away:
+For transformations that [`shift_array`][icechunk.Session.shift_array] and [`roll_array`][icechunk.Session.roll_array] can't express, [`reindex_array`][icechunk.Session.reindex_array] gives you complete control. You provide a function that maps each chunk's old position to its new position.
+
+Your function receives a chunk index (as a list) and returns:
 
-| Value | Behavior |
-|-------|----------|
-| `True` | Vacated positions are deleted (return fill value) |
-| `False` | Vacated positions keep stale references |
+- A new index (as a list) to move the chunk there
+- `None` to discard the chunk
 
 ```python exec="on" session="chunks" source="material-block" result="code"
 repo = ic.Repository.create(ic.in_memory_storage())
@@ -220,25 +174,19 @@ arr = zarr.create(
     fill_value=-1,
 )
 arr[:] = np.arange(10)
-session.commit("setup")
+print("Before:", arr[:])
 
-def shift_left_2(idx):
+def shift_and_filter(idx):
+    """Shift left by 2, discard chunks that would go negative."""
     new_idx = idx[0] - 2
     return None if new_idx < 0 else [new_idx]
 
-# delete_vacated=False: source positions keep stale data
-session = repo.writable_session("main")
-arr = zarr.open_array(session.store, path="arr")
-session.reindex_array("/arr", shift_left_2, delete_vacated=False)
-print("delete_vacated=False:", arr[:])
-
-# delete_vacated=True: vacated positions return fill value
-session = repo.writable_session("main")
-arr = zarr.open_array(session.store, path="arr")
-session.reindex_array("/arr", shift_left_2, delete_vacated=True)
-print("delete_vacated=True: ", arr[:])
+session.reindex_array("/arr", shift_and_filter)
+print("After: ", arr[:])
 ```
 
+Vacated positions retain stale chunk references.
+
 ### Custom Transformations
 
 With `reindex_array`, you can implement any chunk permutation:
@@ -261,7 +209,7 @@ def reverse_chunks(idx):
     """Reverse the order of all chunks."""
     return [4 - idx[0]]  # 0↔4, 1↔3, 2 stays
 
-session.reindex_array("/arr", reverse_chunks, delete_vacated=False)
+session.reindex_array("/arr", reverse_chunks)
 print("After: ", arr[:])
 ```
 
@@ -287,7 +235,7 @@ def swap_quadrants(idx):
     row, col = idx
     return [(row + 1) % 2, (col + 1) % 2]
 
-session.reindex_array("/arr2d", swap_quadrants, delete_vacated=False)
+session.reindex_array("/arr2d", swap_quadrants)
 print("\nAfter swapping quadrants:")
 print(arr[:])
 ```

icechunk-python/python/icechunk/__init__.py

@@ -48,7 +48,6 @@
     S3Options,
     S3StaticCredentials,
     SessionMode,
-    ShiftMode,
     SnapshotInfo,
     Storage,
     StorageConcurrencySettings,
@@ -164,7 +163,6 @@
     "S3StaticCredentials",
     "Session",
     "SessionMode",
-    "ShiftMode",
     "SnapshotInfo",
     "Storage",
     "StorageConcurrencySettings",

icechunk-python/python/icechunk/_icechunk_python.pyi

@@ -1888,20 +1888,6 @@ class SessionMode(Enum):
     WRITABLE = 1
     REARRANGE = 2
 
-class ShiftMode(Enum):
-    """The mode for shifting array chunks, determining how out-of-bounds chunks are handled.
-
-    Attributes
-    ----------
-    WRAP: int
-        Circular buffer - chunks wrap around to the other side, shape unchanged
-    DISCARD: int
-        Out-of-bounds chunks are discarded, vacated positions return fill_value
-    """
-
-    WRAP = 0
-    DISCARD = 1
-
 class PySession:
     @classmethod
     def from_bytes(cls, data: bytes) -> PySession: ...
@@ -1924,11 +1910,9 @@ class PySession:
         self,
         array_path: str,
         shift_chunk: Callable[[Iterable[int]], Iterable[int] | None],
-        delete_vacated: bool,
     ) -> None: ...
-    def shift_array(
-        self, array_path: str, chunk_offset: Iterable[int], mode: ShiftMode
-    ) -> list[int]: ...
+    def shift_array(self, array_path: str, chunk_offset: Iterable[int]) -> list[int]: ...
+    def roll_array(self, array_path: str, chunk_offset: Iterable[int]) -> list[int]: ...
     async def move_node_async(self, from_path: str, to_path: str) -> None: ...
     def all_virtual_chunk_locations(self) -> list[str]: ...
     async def all_virtual_chunk_locations_async(self) -> list[str]: ...