`concat_on_disk` merge strategies are untested/not implemented

[ilan-gold]

Please describe your wishes and possible alternatives to achieve the desired result.

We should implement them so that they work properly. I am not really sure if this is a bug since concat_on_disk is experimental and reading through the old PR, I don't see any discussion of it or any tests for it.

Here's an MVCE for first with default arguments, although by adding a merge_type argument to the current test suite for concat_on_disk you can see a full list of problems:

from anndata.tests.helpers import (
    assert_equal,
    gen_adata,
)
import anndata as ad
import numpy as np
from scipy import sparse
import pandas as pd

GEN_ADATA_OOC_CONCAT_ARGS = dict(
    obsm_types=(
        sparse.csr_matrix,
        np.ndarray,
        pd.DataFrame,
    ),
    varm_types=(sparse.csr_matrix, np.ndarray, pd.DataFrame),
    layers_types=(sparse.spmatrix, np.ndarray, pd.DataFrame),
)

adata_1 = gen_adata((100, 200), **GEN_ADATA_OOC_CONCAT_ARGS)
adata_2 = gen_adata((50, 60), **GEN_ADATA_OOC_CONCAT_ARGS)
adata_1.write_h5ad('test_1.h5ad')
adata_2.write_h5ad('test_2.h5ad')
ad.experimental.concat_on_disk(['test_1.h5ad', 'test_2.h5ad'], 'merged.h5ad', merge="first")

raises:

IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
Error raised while writing key 'var_cat' of <class 'h5py._hl.group.Group'> to /var

Here's a full list of the tests that fail from test_anndatas_with_reindex when merge is tested:

Errors + tests

FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-inner-zarr-10-unique] - AssertionError: Error raised from element 'varp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-inner-h5ad-10-same] - AssertionError: Error raised from element 'varp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-inner-h5ad-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-inner-zarr-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-inner-h5ad-100000000-unique] - AssertionError: Error raised from element 'varp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-inner-h5ad-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-outer-h5ad-100000000-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-inner-zarr-100000000-same] - AssertionError: Error raised from element 'varp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-outer-h5ad-10-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-outer-h5ad-100000000-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-inner-zarr-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-inner-zarr-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-inner-zarr-10-same] - AssertionError: Error raised from element 'varp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-inner-h5ad-100000000-unique] - AssertionError: DataFrame are different
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-inner-h5ad-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-outer-zarr-100000000-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-outer-zarr-10-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-inner-zarr-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-inner-h5ad-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-inner-h5ad-100000000-same] - AssertionError: Error raised from element 'varp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-array-outer-h5ad-10-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-outer-h5ad-10-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-inner-h5ad-10-same] - AssertionError: Error raised from element 'varp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-outer-h5ad-100000000-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-inner-h5ad-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-inner-h5ad-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-inner-h5ad-10-same] - AssertionError: Error raised from element 'obsp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-inner-h5ad-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-inner-h5ad-100000000-same] - AssertionError: Error raised from element 'obsp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-inner-h5ad-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-inner-zarr-100000000-same] - AssertionError: Error raised from element 'obsp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-inner-zarr-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-outer-zarr-100000000-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-inner-zarr-10-unique] - AssertionError: Error raised from element 'obsp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-inner-zarr-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-inner-zarr-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-outer-h5ad-10-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-outer-h5ad-100000000-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-outer-zarr-100000000-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-outer-zarr-10-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-array-outer-zarr-10-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-inner-zarr-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-inner-zarr-100000000-unique] - AssertionError: Error raised from element 'obsp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-inner-h5ad-10-same] - AssertionError: Error raised from element 'obsp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-inner-h5ad-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-inner-zarr-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-inner-h5ad-100000000-same] - AssertionError: Error raised from element 'obsp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-outer-zarr-10-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-inner-h5ad-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse_array-inner-zarr-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-inner-zarr-100000000-unique] - AssertionError: Error raised from element 'varp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-outer-zarr-100000000-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-outer-h5ad-10-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-outer-h5ad-100000000-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-inner-zarr-10-unique] - AssertionError: Error raised from element 'obsp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse_array-inner-zarr-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[0-sparse-outer-zarr-100000000-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse_array-inner-h5ad-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse_array-inner-h5ad-100000000-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'h5py._hl.group.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse_array-inner-h5ad-100000000-unique] - AssertionError: Error raised from element 'obsp'.
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse-outer-zarr-10-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse_array-outer-zarr-10-first] - TypeError: expected unicode string, found True
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse_array-inner-zarr-10-first] - anndata._io.specs.registry.IORegistryError: No method registered for writing <class 'pandas.core.series.Series'> into <class 'zarr.hierarchy.Group'>
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse_array-outer-h5ad-10-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse_array-outer-h5ad-100000000-first] - TypeError: Can't implicitly convert non-string objects to strings
FAILED tests/test_concatenate_disk.py::test_anndatas_with_reindex[1-sparse_array-outer-zarr-100000000-first] - TypeError: expected unicode string, found True

Details

[annavpo]

Hi @ilan-gold
Running the following
ad.experimental.concat_on_disk(
h5ad_dict,
'../merged.h5ad',
label='82_organoid_samples',
)

I get this error:
IORegistryError: No method registered for writing <class 'dask.array.core.Array'> into <class 'h5py._hl.group.Group'>
Error raised while writing key 'X_pca' of <class 'h5py._hl.group.Group'> to /obsm

Has this been solved? I use scanpy==1.10.4, anndata==0.11.3
Thanks!

[ilan-gold]

It has not been solved yet, no. But it looks like you don't have any arguments around merge strategies. So please open an issue with a small reproducible example that I can run locally if you would like help.