Tests take far too long

[flying-sheep]

Please make sure these conditions are met

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of anndata.
• (optional) I have confirmed this bug exists on the master branch of anndata.

Report

We talked about it before, but tests should maybe take around 5 minutes or less, not 25!

I think it’s zarr stuff that takes that long, right? @ilan-gold could you describe why this is again here so we can work on a fix?

Versions

Details

Dependency	Version
wcwidth	0.2.13
Deprecated	1.2.18
toolz	1.0.0
jaraco.context	5.3.0
h5py	3.13.0
PyYAML	6.0.2
stack-data	0.6.3
hatch-vcs	0.4.0
pathspec	0.12.1
scanpy	1.11.0
awkward_cpp	45
jedi	0.19.2
cloudpickle	3.1.1
pytz	2025.2
jaraco.functools	4.0.1
executing	2.2.0
legacy-api-wrap	1.4.1
donfig	0.8.1.post1
fsspec	2025.3.0
zarr	3.0.6
numcodecs	0.15.1
pure_eval	0.2.3
parso	0.8.4
natsort	8.4.0
session-info2	0.1.2
scipy	1.15.2
python-dateutil	2.9.0.post0
pluggy	1.5.0
wrapt	1.17.2
hatchling	1.27.0
numpy	2.1.3
dask	2025.1.0
awkward	2.8.1
coverage	7.7.1
MarkupSafe	3.0.2
typing_extensions	4.13.0
traitlets	5.14.3
Jinja2	3.1.6
pyarrow	19.0.1
crc32c	2.7.1
msgpack	1.1.0
jaraco.text	3.12.1
packaging	24.2
asttokens	3.0.0
pandas	2.2.3
setuptools	78.1.0
ipython	9.0.2
more-itertools	10.3.0
tblib	3.0.0
psutil	7.0.0
rich	13.9.4
prompt_toolkit	3.0.50
decorator	5.2.1
Pygments	2.19.1
six	1.17.0
setuptools-scm	8.2.0
jaraco.collections	5.1.0

Component	Info
Python	3.13.1 (main, Dec 6 2024, 20:13:21) [Clang 18.1.8 ]
OS	macOS-15.5-arm64-arm-64bit-Mach-O
Updated	2025-07-10 17:58

[ilan-gold]

I think it’s zarr stuff that takes that long, right? @ilan-gold could you describe why this is again here so we can work on a fix?

I am experimenting now. There are definitely some tests that were taking upwards of 10 seconds, which is bad, but there were maybe 20 or 30 total. So I brought those times down by a factor of 4X or so. But that made only about a ~12% dent locally.

Separately we simply have a lot of tests, over 4000, and specifically re: zarr, we have a lot of tests for both v2 and v3. I suspect bringing down the number of v2/v3 tests would help a bit. So we'd have to go through every test and ask "do we really need a test for both v3 and v2 zarr."

I also have a suspicion that dask/dask#11952 is having a field day with us. If it takes half of a second to generate a dask array via map_blocks, that won't pop up in any "what is the longest test?" profiling but would make a dent overall. I will try locally to see if my fix for that helps or not.

These are just first thoughts.

[ilan-gold]

Another culprit would be the zarr package itself having overhead per-write. The minimum deps job passes in 8 minutes despite running a comparable number of tests.

[ilan-gold]

Another observation (re: the above), I didn't see any difference between time taken with an in-memory store vs. on-disk (in fact, memory was worse unless I didn't see that right).

[ilan-gold]

TODO tomorrow: run tests for zarr in a separate job

[flying-sheep]

in fact, memory was worse unless I didn't see that right

that part is pretty weird, otherwise this all makes sense to me!

when adding the split job, we should add a comment mentioning that once that job runs in <5 minutes or so, we should merge the jobs back together.