harpy.im.pixel_clustering_preprocess with scale_factors error

[nbonine]

When trying to run harpy.im.pixel_clustering_preprocess with the option scale_factors, I get the following error message:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[12], line 1
----> 1 sdata = hp.im.pixel_clustering_preprocess(
      2     sdata,
      3     image_name = img_name,
      4     output_image_name = f"{img_name}_px_preprocessed",

File c:\Users\nbonine\harpy_env_project\.venv\Lib\site-packages\harpy\image\pixel_clustering\_preprocess.py:254, in pixel_clustering_preprocess(sdata, image_name, output_image_name, channels, p, p_sum, p_post, sigma, norm_sum, cap_max, chunks, scale_factors, cast_dtype, persist_intermediate, overwrite)
    252 # save the preprocessed images, in this way we get the preprocessed images from which we sample
    253 for i in range(len(_arr_list)):
--> 254     sdata = add_image(
    255         sdata,
    256         arr=_arr_list[i].squeeze(1) if to_squeeze else _arr_list[i],
    257         output_image_name=output_image_name[i],
    258         transformations=_transformations[i],
    259         scale_factors=scale_factors,
    260         c_coords=channels,
    261         overwrite=overwrite,
    262     )
    264 if clean_up:
    265     # clean up the intermediate zarr store.
    266     for i in range(len(_arr_list)):

File c:\Users\nbonine\harpy_env_project\.venv\Lib\site-packages\harpy\image\_image.py:290, in add_image(sdata, arr, output_image_name, dims, chunks, transformations, scale_factors, c_coords, overwrite)
...
--> 121 if group.metadata.zarr_format == 3 and len(multiscales := group.metadata.attributes["ome"]["multiscales"]) != 1:
    122     len_scales = len(multiscales)
    123     raise ValueError(f"The length of multiscales metadata should be 1, found the length to be {len_scales}")

KeyError: 'ome'

The code runs perfectly without using the scale_factors

This could be related to a more general issue of writing multiscale elements with a dask client (as reported in scverse/spatialdata#1024), not sure.

[ArneDefauw]

Hi @nbonine , are you using a dask client when setting scale_factors=... ?

If so, yes, this will be related to scverse/spatialdata#1024.

If you need the preprocessed results as multiscale, and you want to use a dask client for the preprocessing, what you could do is the following:

run hp.im.pixel_clustering_preprocess with a dask client, and with scale_factors=None, then close the dask client, and rewrite the preprocessed image again, but now multiscale, e.g.:

hp.im.pixel_clustering_preprocess(sdata, image_name="image", output_image_name="image_preprocessed", scale_factors=None,...)
client.close()

hp.im.add_image(
sdata,
arr=sdata["image_preprocessed"],
scale_factors=[2,2,2,2],
output_image_name="image_preprocessed",
c_coords=sdata["image_preprocessed"].c.data,
overwrite=True,
)

(This is how I circumvent the bug typically. I will report the bug upstream, I think it is a bug in the ome-zarr library. )

[nbonine]

Hi @ArneDefauw ,

Indeed I was using a dask client.
It would indeed be great to report this upstream. Thanks for the workaround!