Skip to content

vesuvius.predict fails on zarr 3 (a supported version) and exits 0 while writing nothing #1360

Description

@TAUIL-Abd-Elilah

vesuvius.predict cannot write its output store under zarr 3, and the failure is silent to
anything that checks a return code. pyproject.toml declares zarr>=2.18.7,<4, so zarr 3 is
supported on paper.

Reproduce

zarr 3.2.1, any model and input:

vesuvius.predict --model_path <model> --input_dir <ct.zarr> --output_dir out \
  --device cuda --disable_tta --batch_size 1 --num_workers 2 --bbox 32:160,32:160,32:160
An error occurred during inference: module 'zarr' has no attribute 'Blosc'
--- Inference Failed ---
Error: cannot unpack non-iterable NoneType object

Exit code is 0 and out/ is empty.

Cause 1 — zarr.Blosc was removed in zarr 3

models/run/inference.py::_get_zarr_compressor (four call sites, lines ~822-830) uses
zarr.Blosc, which is a zarr-2 API.

The fix looks like a one-liner because on zarr 2 that name is literally the numcodecs class:

>>> import zarr, numcodecs; zarr.__version__
'2.18.3'
>>> zarr.Blosc is numcodecs.Blosc
True

So importing Blosc from numcodecs is a no-op on zarr 2 and a fix on zarr 3.

Cause 2 — the store is then created as v3 with a v2-only argument

With Blosc resolved, the next failure is:

ValueError: compressor cannot be used for arrays with zarr_format 3.
Use bytes-to-bytes codecs instead.

data/utils.py:144 passes compressor= to zarr.open without naming a format, so zarr 3
defaults to v3, where compressor is not a valid argument. Either pin zarr_format=2 when a
compressor is supplied, or translate to codecs=[BloscCodec(...)] for v3. Pinning v2 reproduces
what zarr 2.18.3 did, which is the combination existing artifacts were produced under.

I worked around both locally by shimming rather than patching, so I have not opened a PR — happy
to if you would prefer a specific direction on cause 2 (v2 pin vs. real v3 codecs), since that
one is a behaviour choice about what the output store should be, not just a compatibility fix.

Cause 3 — separate, and worse for pipelines: failure exits 0

inference.main() and blending.main() both catch their own exceptions, print
--- ... Failed ---, and return normally. Measured on the run above:

REAL EXIT CODE: 0
--- output dir contents:      (empty)

Anything driving these as subprocesses — a whole-scroll sweep, a benchmark harness, CI — sees
success and proceeds on missing data. I hit this from the other end: my wrapper checked
returncode, sailed past a dead inference step, and then died much later on a missing
merged.zarr, with a traceback pointing at the wrong stage entirely.

Worth a sys.exit(1) on those paths regardless of how causes 1 and 2 are resolved.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions