Show tqdm progress when loading large datasets

[igor-sadalski]

Please describe your wishes and possible alternatives to achieve the desired result.

When loading large datasets to RAM memory (e.g. 500GB dataset) it takes around 30 minutes. It would be useful to show progress with tqdm and estimate time it would take

[ilan-gold]

@igor-sadalski If all of the big data is in X there's not really a path forward on that.

Have you tried https://anndata.readthedocs.io/en/stable/generated/anndata.experimental.read_lazy.html or https://anndata.readthedocs.io/en/stable/generated/anndata.io.sparse_dataset.html for your downstream task, especially if you're using scanpy or smaller in-memory subsets or just exploring your data?

If all your data is in X a tqdm bar wouldn't do much because most of the time is spent in a_zarr_or_h5_array[...] and those libraries don't provide progress bars to my knowledge. And I don't think a progress bar that goes per anndata-element makes much sense given how much time is spent disproportionately (I think) in only a few calls like a_zarr_or_h5_array[...].

Do you have a suggestion?