CZI Opening with Bioformats Get Region Hangs

[amrosado]

I'm trying to use getRegionAtAnotherScale with CZI files. It works fine at first, but stalls when trying to read certain regions. Is there a certain JVM tested for with CZI files that I should ensure is being used?

Is it possible to install the development openslide and force the CZI file to be read with the opneslide tile source? I see that the priority for CZI is undefined given the last published release was in 2023 which did not support CZI, but I would like to be able to test with openslide dev if possible.

[manthey]

I noticed that when we use bioformats as a tile source that we were not quite locking where we needed to be (see #1896) which specifically, made reading multiple tiles slower than it could be and had the chance that the javabridge could fail to do proper garbage collection. This change might fix what you are seeing (or it might do nothing).

If you install on a linux machine with the large_image_wheels, we include a jvm as part of that which is what gets tested. At least in our tests, I don't know of any czi files which fail (which could just mean we don't have a representative file in our tests).

The openslide library can read uncompressed czi files. There is a PR on openslide that adds support for jp2k compressed czi files (the most common variant I've seen). One could build the openslide library with that PR enabled and see if it works.

I'll be merging #1896; can you see if that resolves your current issue? If not, can you share an image and what region fails?

[amrosado]

Sounds good! I can test large_image with CZI files with the new version and if there are still issues maybe I can help with getting more comprehensive testing for CZI.

[manthey]

The python-bioformats package required some of the locking I removed in #1896, so I had to add back some of it in #1898. I'm certain the python-bioformats could be made more thread safe to allow better parallelism, but that isn't something I have inclination to dig into. I'm still interested in what has landed will resolve your use case.

[manthey]

I'm closing this; bioformats is slow but shouldn't hang. We can reopen or have another issue if there is something else that is actionable.

[amrosado]

I've tried with the latest changes. It still hangs part way for getting regions. This is in the context of the eager iterator. I would appreciate your help figuring out this issue.

[manthey]

When it is hung, can you do pystack remote <pid> of the stuck process and see where it is stuck?

[amrosado]

Okay, sounds good. I'll give that a try to see if it elucidates anything I can change. My feeling is that it is some issue with the java bridge. Any recommendations for debugging the java bridge?

[manthey]

Yes, my expectation is also that pystack will show that it is in the java side of things. I would expect if that is the case it should be reproducible with just a specific query. The python-bioformats package has a fair amount of code to process what comes back from java; it reuses a buffer to fetch data from java to python and isn't thread safe because of that; I wonder if there is some locking going on with that buffer that I haven't considered, in which case you might see where something is waiting for a lock from pystack.