create_empty_plate shard/chunk geometry forces read-modify-write, making torn shards recur

[tayllatheodoro]

Tracking issue from czbiohub-sf/biahub#286 (recurrent torn-shard corruption on the mantis-v2 pipeline). Related to #415 and zarrs/zarrs#420.

Plate outputs created via create_empty_plate use a geometry that forces a read-modify-write on every per-(t,c) write, which is why a single torn/partially-written shard fails deterministically on every subsequent read or write (so -resume and retries can never get past it).

Verified from the array zarr.json

• array shape [67, 3, 1068, 256, 1664] (T, C, Z, Y, X)
• shard (top-level chunk_grid) [1, 1, 1072, 256, 1792] → one shard per (t, c)
• inner chunk [1, 1, 16, 256, 256]

Z=1068 is not a multiple of the inner z-chunk 16, and X=1664 is not a multiple of 256, so the boundary inner chunks are only partially covered by each write. zarr must therefore read the existing shard and merge (RMW). Once a shard is torn — e.g. a job killed mid-write, exactly the case @LDeakin described in #415 — every subsequent write re-reads it and raises either:

• RuntimeError: the checksum is invalid, or
• RuntimeError: The encoded shard is smaller than the expected size of its index.

Both surface from the same path: iohub/ngff/utils.py: apply_transform_to_tczyx_and_save → _save_transformed → write_oindex → set_orthogonal_selection → zarrs/pipeline.py: write.

The geometry originates where the plate is created (create_empty_plate, chunks=None defaulting to (1,1,16,256,256), with the shard extent rounded up past the data extent).

Proposals

• Choose inner-chunk sizes that divide the data dims (or a shard extent equal to the data extent) so each (t,c) write fully covers its inner chunks → no RMW read-back → a torn shard from a prior kill is overwritten wholesale instead of re-read. This would largely eliminate the recurrence even under preemption.
• Track/adopt upstream graceful handling of partial/corrupt existing shards (Add storage adapter that writes a tmp file then atomically moves on success zarrs/zarrs#420).

Related

• Upstream discussion of the root cause (killed-job partial writes): RuntimeError: the checksum is invalid when writing zarr chunks via zarrs codec on Lustre #415
• zarrs graceful-overwrite: Add storage adapter that writes a tmp file then atomically moves on success zarrs/zarrs#420
• Downstream mitigation (per-position self-heal): feat(nextflow): self-heal torn-shard/IO failures on per-position steps + default to non-preemptible cpu partition biahub#285