Neighborhood matrix ValueError: did not find a match in any of xarray's currently installed IO backends ['scipy', 'zarr'].

[smith6jt-cop]

Please refer to our FAQ and look at our known issues before opening a bug report.

Describe the bug
Running in Ubuntu 24.04.1 LTS, ark-analysis-0.7.3.dev0+ge22a487.d20241011 in conda env, example_neighborhood_analysis.ipynb, cell 8

ValueError Traceback (most recent call last)
Cell In[8], line 8
4 neighbor_freqs = pd.read_csv(freqs_path)
6 else:
7 # Create new matrix with the radius and cell column specified above
----> 8 neighbor_counts, neighbor_freqs = neighborhood_analysis.create_neighborhood_matrix(
9 all_data, dist_mat_dir, distlim=pixel_radius, cell_type_col=cell_type_col)
11 # Save neighbor matrices
12 neighbor_counts.to_csv(counts_path, index=False)

File ~/miniconda3/envs/ark_env/lib/python3.10/site-packages/ark/analysis/neighborhood_analysis.py:92, in create_neighborhood_matrix(all_data, dist_mat_dir, included_fovs, distlim, self_neighbor, fov_col, cell_label_col, cell_type_col)
89 fov_cluster_names = current_fov_neighborhood_data[cell_type_col].drop_duplicates()
91 # Retrieve fov-specific distance matrix from distance matrix dictionary
---> 92 dist_matrix = xr.load_dataarray(os.path.join(dist_mat_dir, str(fov) + '_dist_mat.xr'))
94 # Get cell_neighbor_counts and cell_neighbor_freqs for fovs
95 counts, freqs = spatial_analysis_utils.compute_neighbor_counts(
96 current_fov_neighborhood_data, dist_matrix, distlim, self_neighbor,
97 cell_label_col=cell_label_col, cluster_name_col=cell_type_col)

File ~/miniconda3/envs/ark_env/lib/python3.10/site-packages/xarray/backends/api.py:303, in load_dataarray(filename_or_obj, **kwargs)
300 if "cache" in kwargs:
301 raise TypeError("cache has no effect in this context")
--> 303 with open_dataarray(filename_or_obj, **kwargs) as da:
304 return da.load()

File ~/miniconda3/envs/ark_env/lib/python3.10/site-packages/xarray/backends/api.py:769, in open_dataarray(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
611 def open_dataarray(
612 filename_or_obj: str | os.PathLike[Any] | BufferedIOBase | AbstractDataStore,
613 *,
(...)
629 **kwargs,
630 ) -> DataArray:
631 """Open an DataArray from a file or file-like object containing a single
632 data variable.
633
(...)
766 open_dataset
767 """
--> 769 dataset = open_dataset(
770 filename_or_obj,
771 decode_cf=decode_cf,
772 mask_and_scale=mask_and_scale,
773 decode_times=decode_times,
774 concat_characters=concat_characters,
775 decode_coords=decode_coords,
776 engine=engine,
777 chunks=chunks,
778 cache=cache,
779 drop_variables=drop_variables,
780 inline_array=inline_array,
781 chunked_array_type=chunked_array_type,
782 from_array_kwargs=from_array_kwargs,
783 backend_kwargs=backend_kwargs,
784 use_cftime=use_cftime,
785 decode_timedelta=decode_timedelta,
786 **kwargs,
787 )
789 if len(dataset.data_vars) != 1:
790 raise ValueError(
791 "Given file dataset contains more than one data "
792 "variable. Please read with xarray.open_dataset and "
793 "then select the variable you want."
794 )

File ~/miniconda3/envs/ark_env/lib/python3.10/site-packages/xarray/backends/api.py:569, in open_dataset(filename_or_obj, engine, chunks, cache, decode_cf, mask_and_scale, decode_times, decode_timedelta, use_cftime, concat_characters, decode_coords, drop_variables, inline_array, chunked_array_type, from_array_kwargs, backend_kwargs, **kwargs)
566 kwargs.update(backend_kwargs)
568 if engine is None:
--> 569 engine = plugins.guess_engine(filename_or_obj)
571 if from_array_kwargs is None:
572 from_array_kwargs = {}

File ~/miniconda3/envs/ark_env/lib/python3.10/site-packages/xarray/backends/plugins.py:197, in guess_engine(store_spec)
189 else:
190 error_msg = (
191 "found the following matches with the input file in xarray's IO "
192 f"backends: {compatible_engines}. But their dependencies may not be installed, see:\n"
193 "https://docs.xarray.dev/en/stable/user-guide/io.html \n"
194 "https://docs.xarray.dev/en/stable/getting-started-guide/installing.html"
195 )
--> 197 raise ValueError(error_msg)

ValueError: did not find a match in any of xarray's currently installed IO backends ['scipy', 'zarr']. Consider explicitly selecting one of the installed engines via the engine parameter, or installing additional IO dependencies, see:
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
https://docs.xarray.dev/en/stable/user-guide/io.html

Expected behavior
Expected for neighborhood matrix to be created without error.

To Reproduce

[srivarra]

@smith6jt-cop Could you try installing netCDF4 in the Ark environment with pip install netCDEF4 and see if that fixes the issue?

[smith6jt-cop]

Hello and thank you for your assistance. I installed with pip install netCDF4 and received a similar error:

ValueError: did not find a match in any of xarray's currently installed IO backends ['netcdf4', 'scipy', 'zarr']. Consider explicitly selecting one of the installed engines via the engine parameter, or installing additional IO dependencies, see:
https://docs.xarray.dev/en/stable/getting-started-guide/installing.html
https://docs.xarray.dev/en/stable/user-guide/io.html

I installed other packages suggested in the xarray documentation. This led to the np.float error in the lda package. Fortunately, there is only one instance of this in all of that packages' scripts so changing from 'np.float' to 'float' solved the problem.

I found that the problem in my case was that I had no files for my dist_matrix and so the plugins.engine could not guess the engine based on the dataset. The reason for this is that in example_neighborhood_analysis cell 21, the function is within an "if not" condition and I had already created the directory. Also, misc_utils.verify_in_list deals with empty lists by checking if the "left hand list" or first list is empty and returning "True" if so:

rhs_name, lhs_name = map(lambda s: s.replace("_", " "), kwargs.keys())

lhs_list, rhs_list = map(lambda l: list(make_iterable(l)), kwargs.values())

# Check if either list inputs are `None` (or both)
if any(v == [None] for v in (lhs_list, rhs_list)):
    return True
# If the Left Hand List is empty, the Right Hand List will always be contained in it.
if len(lhs_list) == 0:
    return True
# If the Right Hand List is empty, the Left Hand List will never be contained in it.
# Throw a ValueError
if len(rhs_list) == 0:
    raise ValueError(f"The list {lhs_name} is empty.")

I suggest you move the directory code for dist_mat_dir up to cell 4 with the other lines related to checking and creating directories, and rethink the usage of misc_utils.verify_in_list.