Question regarding updating adata.obsm

[ruochiz]

Hi,

Thanks for creating this great package for on-disk anndata. I have a question regarding the obsm entry of the data. Is it possible to do a batch update or batch creation for the obsm in python? Sth like

b = {'a': array1, 'b': array2....}
adata.obsm.update(b)

I did a quick benchmarking of adata.obsm = b and a for loop that create each entry of b in adata.obsm, the former is much faster. However, it doesn't allow overlapping update (like b and adata.obsm have some overlapping keys, and for those, overwrite them with b) or just non-overlapping batch update. Any insights on how to achieve that would be greatly appreciated, thx.

[kaizhang]

@ruochiz Thanks for reporting this issue. Could you please share a little more about the benchmark details? For example, what is the size of the for loop? What's the running time of each strategy? The difference here may be that adata.obsm = b uses Rust's for loop and there is only one FFI call. But I would be surprised if there is a large difference for relatively small updates, e.g., < 100 entries.

[ruochiz]

Thanks for the quick reply. Yeah, I might be abusing the structure of anndata a bit, with a lot of obsm entries (can go up to 100k entries, hence favoring batch update + on-disk anndata combination). Size of each entry would be around 2 x 6 x 1000 array.

Details of benchmarking:

Yeah, I have a similar hypothesis that it's either py -> rust data type conversion, or sth related to flush cache into hdf5 file system (based on the IO, putting everything at once triggers a xxMb/s writing while the later is constantly xxKb/s writing. Wonder if that would means multiple lock / unlock file operation as well. )

[kaizhang]

This is mainly Python -> Rust overhead. I will add an update method to obsm and others. Thanks!

[ruochiz]

Thank you!

[kaizhang]

@ruochiz It doesn't seem to make a big difference in my hand. Here is the code I used:

import anndata_rs as ad
import numpy as np
import time

matrices = {str(i): np.random.rand(2,6,1000) for i in range(10000)}

data = ad.AnnData(filename='data.h5ad')

print("batch update")
t1 = time.time()
data.obsm = matrices
t2 = time.time()
print("take %f seconds" % (t2 - t1))

data.close()

data = ad.AnnData(filename='data.h5ad')

print("for loop update")
t1 = time.time()
for k, m in matrices.items():
    data.obsm[k] = m
t2 = time.time()
print("take %f seconds" % (t2 - t1))

Output:

batch update
take 2.828825 seconds
for loop update
take 3.277861 seconds

[ruochiz]

Hah. I think I know what I did, it turns out for my test code above the original adata.obsm is not removed after batch update, and when doing for loop, it takes more time to delete the existing key.

I suppose I made the same mistake in the project I'm working on where I update the same key couple times. It doesn't take much extra time to update in-memory dict, but for on-disk anndata, it surely makes a difference when deleting.

For the above example, it takes ~30s to delete and replace the obsm.

[kaizhang]

Hah. I think I know what I did, it turns out for my test code above the original adata.obsm is not removed after batch update, and when doing for loop, it takes more time to delete the existing key.

I suppose I made the same mistake in the project I'm working on where I update the same key couple times. It doesn't take much extra time to update in-memory dict, but for on-disk anndata, it surely makes a difference when deleting.

For the above example, it takes ~30s to delete and replace the obsm.

I'm wondering why deleting data takes so long. Do you have a code example for me to reproduce what you just mentioned?

[ruochiz]

Hey, sth as simple as repeating the above batch update would trigger that

import snapatac2 as ad
import numpy as np
import time

matrices = {str(i): np.random.rand(2,6,1000) for i in range(10000)}

data = ad.AnnData(filename='data.h5ad')
for i in range(5):
    print("batch update: trial %d" % (i+1))
    t1 = time.time()
    data.obsm = matrices
    t2 = time.time()
    print("take %f seconds" % (t2 - t1))


data.close()

data = ad.read('data.h5ad')
print("close + reopen seems to fix it")
t1 = time.time()
data.obsm = matrices
t2 = time.time()
print("take %f seconds" % (t2 - t1))
data.close()

output

batch update: trial 1
take 2.671558 seconds
batch update: trial 2
take 13.101982 seconds
batch update: trial 3
take 12.526524 seconds
batch update: trial 4
take 12.594909 seconds
batch update: trial 5
take 12.409336 seconds
close + reopen seems to fix it
take 3.499502 seconds

For for-loop update

import snapatac2 as ad
import numpy as np
import time

matrices = {str(i): np.random.rand(2,6,1000) for i in range(10000)}

data = ad.AnnData(filename='data.h5ad')
for i in range(5):
    print("for loop update: trial %d" % (i+1))
    t1 = time.time()
    for k, m in matrices.items():
        data.obsm[k] = m
    t2 = time.time()
    print("take %f seconds" % (t2 - t1))


data.close()

data = ad.read('data.h5ad')
print("close + reopen seems to fix it?")
t1 = time.time()
data.obsm = matrices
t2 = time.time()
print("take %f seconds" % (t2 - t1))
data.close()

for loop update: trial 1
take 2.398951 seconds
for loop update: trial 2
take 31.619421 seconds
for loop update: trial 3
take 31.640539 seconds
for loop update: trial 4
take 31.357190 seconds
for loop update: trial 5
take 31.214165 seconds
close + reopen seems to fix it?
take 3.603333 seconds