Skip to content

shift_array interaction with empty chunks #1862

@ianhi

Description

@ianhi

If you shift an empty chunk onto a filled chunk then the filled chunk will (I believe incorrectly) retain it's original value.

Consdier:

Fill value of -1 an array with only the middle element written (chunksize=1)

[-1, 2, -1]

now shift by (1,)

you currently get
[-1, 2, 2]

but I think you should get [-1, -1, 2]

import icechunk as ic
import zarr

repo = ic.Repository.create(storage=ic.in_memory_storage())
session = repo.writable_session("main")
arr = zarr.open_array(
    session.store,
    path="data",
    mode="w",
    shape=(3,),
    chunks=(1,),
    dtype="f4",
    fill_value=-1,
)
arr[1] = 2.0  # only c/1 written; c/0 and c/2 are fill
session.commit("initial")

print("array with only item 1 filled - fill vlaue=-1")
print("before shift:", arr[:])  # [-1, 2, -1]

"""Sparse array shift: only c/1 written, c/0 and c/2 never written. Shift +1."""
print("shift (1,)")
session = repo.writable_session("main")
arr = zarr.open_array(session.store, path="data")
session.shift_array("/data", (1,))

print("after shift: ", arr[:])  # expect [-1, -1, 2] — c/1 vacated, c/2 gets c/1's data
# icechunk gives [-1, 2, 2] — c/1 keeps stale data

Metadata

Metadata

Assignees

Labels

bug 🐛Something isn't working

Type

No type

Projects

Status

No status

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions