Failing tests in test_bioformats.py

[GenevieveBuckley]

Tests are failing in test_bioformats.py. I noticed this while working on PR #403, but it seems to happen for all branches, so I don't think it's caused by my code.

To create a test environment, including dependencies for video files:

conda create -n pims-dev python=3.9 pip ipython
conda activate pims-dev
cd GitHub/pims
python -m pip install -r requirements-dev.txt
python -m pip install -e .
conda install -c conda-forge av imageio-ffmpeg moviepy tifffile

To run the tests:

python download_bioformats_test.py
pytest pims/tests/test_bioformats.py -vv

Problems encountered:

• Inconsistent failing tests. In test_bioformats.py I see a lot of test_repr failures, but when I run just the single test it passes fine. Commenting out test_repr leads to other tests failing (that again, pass fine when run individually). Is something weird happening where the test files are not being closed properly? I don't know how to debug this.
• There's something wrong with the norpix reader. It can't open the HEART.SEQ test image file. Oddly, all of the tests in pytest pims/tests/test_norpix.py pass. But pytest pims/tests/test_bioformats::TestBioformatsSEQ::test_open fails.

More weird stuff I have noticed:

• Only the PyAV reader can open the file pims/tests/data/bioformats/wtembryo.mov. If PyAV is not available, sometimes other handlers try and fail (which casues a test failure). Given that this file will play in VLC but not QuickTime, I assume that's something to do with the codecs? I don't think this is a problem, because at least something can open the file, but it does mean you must have PyAV installed as a dependency for the tests not to fail.

[nkeim]

@GenevieveBuckley Given that the other tests are passing now, is it worth checking this again before we proceed with #403 ?

[GenevieveBuckley]

Some good news, we're down to one failing test as far as I can tell.

Result of pytest pims/tests/test_bioformats.py -vv
TestBioformatsSEQ.test_open is the one currently failing.

pytest report:

=================================== FAILURES ===================================
_________________________ TestBioformatsSEQ.test_open __________________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x13e3bede0>
when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.

        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

../../../mambaforge/envs/pims/lib/python3.11/site-packages/_pytest/runner.py:339:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../../mambaforge/envs/pims/lib/python3.11/site-packages/_pytest/runner.py:260: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
../../../mambaforge/envs/pims/lib/python3.11/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
../../../mambaforge/envs/pims/lib/python3.11/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
../../../mambaforge/envs/pims/lib/python3.11/site-packages/_pytest/unraisableexception.py:88: in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO [closed]>
E
E               Traceback (most recent call last):
E                 File "/Users/genevieb/Documents/GitHub/pims/pims/api.py", line 203, in open
E                   messages.append('{} errored: {}'.format(str(handler), str(e)))
E               ResourceWarning: unclosed file <_io.BufferedReader name='/Users/genevieb/Documents/GitHub/pims/pims/tests/data/bioformats/HEART.SEQ'>

../../../mambaforge/envs/pims/lib/python3.11/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning
=========================== short test summary info ============================
FAILED pims/tests/test_bioformats.py::TestBioformatsSEQ::test_open - pytest.PytestUnraisableExceptionWarning: Exception ignored in: <_io.FileIO ...
======================== 1 failed, 169 passed in 35.74s ========================

[nkeim]

Hooray! I think I have an idea of what's unusual about TestBioformatsSEQ.test_open: calling pims.open('HEART.SEQ') will first try the built-in reader for the StreamPix format, which is an entirely different format that coincidentally has a .seq extension. I get

>>> v = pims.open('HEART.SEQ')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/nkeim/projects/trackpy/pims/pims/api.py", line 200, in open
    "All handlers returned exceptions:\n" + "\n".join(messages))
pims.api.UnknownFormatError: All handlers returned exceptions:
<class 'pims.norpix_reader.NorpixSeq'> errored: The format of this .seq file is unrecognized
<class 'pims.bioformats.BioformatsReader'> errored: No JVM shared library file (libjli.dylib) found. Try setting up the JAVA_HOME environment variable properly.
>>> v
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'v' is not defined

I included the second prompt to point out that the assignment fails. So, here's my hypothesis, following the definition in test_bioformats.py of _image_single.test_open() on line 38, which is

    def test_open(self):
        self.v.close()
        self.v = pims.open(self.filename)

1. pims.open('HEART.SEQ') tries the internal reader, which fails.
2. Then, for some reason, the Bioformats reader also fails? Or maybe the failure of the internal reader is enough to cause the problem.
3. self.v has not been replaced, which means that it still references a closed file object.
4. This creates a problem for some unspecified later code, which assumes that either self.v is undefined, or it is an open file.
5. Since this problem happens outside the test, there is a pytest.PytestUnraisableExceptionWarning.

My proposed change is:

    def test_open(self):
        self.v.close()
        del self.v
        self.v = pims.open(self.filename)

Then, hopefully, we can see what originally caused the problem.

What do you think? (I've had bad experiences with installing the Java SDK on my Mac so I've been trying to avoid it.)

[GenevieveBuckley]

Worth trying, I guess!

[nkeim]

My idea did not work 😞. But I can confirm that the exception is raised when attempting some operation on the HEART.SEQ file after it has been closed — not sure whether it's the bioformats or pims reader that's responsible.

Given that this is a niche format and so far it's just the test that fails, I'm going to set this aside for now.

[nkeim]

I think I found the problem: After it decides the format is invalid, the built-in Norpix reader doesn't close the file. Ordinarily this is no problem because the failed reader will be garbage-collected before most other things can happen. But during tests there may not always be enough time.

Adding a __del__() method that closes the file seems to work. For good measure I also added one in cine.py. I will submit a PR soon.

[GenevieveBuckley]

Nice work! Thank you for your perseverance 🎉