Are trailing partial chunks padded?

[maxrjones]

I'm having difficulties with the default ArrayBytesCodec for the last partial chunk of a dataset. Upon inspection, it seems like Zarr-Python pads any trailing partial chunks. Is this interpretation correct? If so, is it intentional? I'm asking because this expectation for all chunks to be complete seems to cause issues for virtualization from other file formats and will also presumable cause errors for any future virtual array concatenation of Zarr stores (e.g., zarr-developers/zarr-specs#288).

Here's an example:

import zarr
from zarr.storage import LocalStore

# Create an array with one full chunk of shape (3,4) and one partial chunk (1,4)
shape = (4,4)
chunks = (3,4)
new_dtype = "uint8"
overwrite=True
zarr_format=3

store = LocalStore(root = ".vscode/zarr-data/example.zarr", read_only=False)
arr = zarr.create_array(
    store,
    name="0",
    shape=shape,
    chunks=chunks,
    dtype=new_dtype,
    zarr_format=3,
    compressors=None,
    filters=None,
    overwrite=overwrite
)
arr[:] = 42

Inspect the size of the two chunks on disk

ls -l .vscode/zarr-data/example.zarr/0/c/0/0 | awk '{print $5}'
# 12 (expected)
ls -l .vscode/zarr-data/example.zarr/0/c/1/0 | awk '{print $5}'
# 12 (I would expect 4)

[d-v-b]

Is this interpretation correct? If so, is it intentional?

for the default chunk grid, yes to both questions.

This can still work for virtualization but only if byte ranges are addressible in the virtualization scheme, and the byte range for all the boundary chunks has been calculated.

[maxrjones]

Interesting, thanks for the link and explanation.

This can still work for virtualization but only if byte ranges are addressible in the virtualization scheme, and the byte range for all the boundary chunks has been calculated.

Hmm I think that alone is not sufficient. For example, I have the addressable byte ranges for boundary chunks but encounter a reshape error in the ArrayBytes codec because the buffer length is less than a full chunk's buffer length. IIUC one option is to add a BytesBytes (e.g., Pad) codec in between any compressors and the ArrayBytes codec that pads the buffer to match a full chunk and make sure that the Pad codec aligns well with the defined ArrayBytes codec so that the correct bytes are truncated. Seems a bit risky though. Do you know off-hand where in the codec pipeline the truncation happens?

An alternative would be to propose a chunk grid extension, but that risks limiting interoperability.

[maxrjones]

I guess we could also define a custom ArrayBytes codec that pads before reshaping, again not sure if this is way too hacky though.

[d-v-b]

ah and I was wrong about the byte addressing thing -- on the encoding side, I think partial chunks are padded to full size before the codec pipeline runs? and if so, there's no way a byte range can be helpful, because the entire padded chunk will be compressed.

[rabernat]

Note that this is the same issue as #3035.

Here's the explanation for this behavior that I provided there.

it makes sense for the following reason: if every chunk is exactly the same size, then we can easily resize the array without ever having to rewrite chunks. Otherwise, Zarr would have to keep explicitly keep track of the size of the chunks somewhere.

For the default chunk grid, every chunk is identical in terms of how it is stored. There is nothing special about the final chunk.