iohub convert fails due to chunk size constraints

[ieivanov]

When calling

iohub convert -i /hpc/instruments/cm.mantis/2026_01_14_first_light/argolight_bullseye_1/ -o /hpc/projects/comp.micro/mantis/2026_01_14_first_light/0-convert/argolight_bullseye_1.zarr

a dask performance warning is raise; conversion proceeds, but parts of the data in the right side is missing - it is set to zeros. Changing the chunking to XY (instead of XYZ) allows the conversion to proceed correctly.

[ieivanov]

Oh, I think I wasn't using the latest version of iohub, that seems to resolve it. Maybe double check to confirm before closing?

[srivarra]

@ieivanov Let me take a look and get back to ya

[srivarra]

@ieivanov I've also ran into the following bugs when running the conversion as is on the latest iohub:

• The image is set to zeros from $X \ge 1392$
• The image is completely set to zeros for all $X$, $Y$ from $Z\ge238$

Changing the chunking from ZYX to YX does resolve both of these for me...

Let me see what else I can find, and maybe figure out why.

[srivarra]

@ieivanov After some tinkering around here's what I've found.

The TIFF files are read slice-by-slice, where each Z is loaded independently, so the source dask array has chunks of shape (1, 1, 1, Y, X). When converting to ZYX chunking (grouping multiple Z's together), the rechunk operation dask_data.rechunk((1, 1, Z, Y, X)) must combine many small source chunks (individual Z slices) into larger target chunks (Z-stacks).

Dask doesn't always do this in a single step. Based on memory constraints set by the array.chunk-size config (the default is ~128MB), it can use intermediate chunks. If the intermediate size doesn't divide evenly into the target, Dask's to_zarr loses data at chunk boundaries therefore filling those regions with zeros.

So the fix is setting dask.config.set({"array.chunk-size": <target_chunk_bytes>}) before rechunking. This tells Dask to use chunks large enough to align with the target if it needs to, avoiding the misaligned intermediate steps that caused that loss.