Use inline backend or Plots pane to display plots

[OverLordGoldDragon]

Matplotlib figures, if not closed, prevent testing (or any test code execution) from advancing. When running pytest via pytest.main([__file__, "-s"]) (F5, __main__), plots are inlined and there isn't a problem. Plots may be part of testing, so silencing isn't always an option.

Moving plots to the Plots pane, or inline, or really anywhere as long as they don't stall testing would work. Is this currently doable?

[jitseniesen]

What happens if you run pytest outside Spyder? In other words, is this problem due to how pytest and matplotlib works or is it due to something we do in Spyder and this plugin?

[OverLordGoldDragon]

@jitseniesen All tests pass, though no plots generated when ran via Anaconda Powershell Prompt. Tests also pass if the test .py file is ran from within Spyder via "Run" as __main__, and plots are generated inline. Only spyder-unittest generates plots in a new window and requires them to be closed before advancing the test.

[jitseniesen]

Thanks. It looks like it is up to us then. Can you please give a minimal (or at least small) example of a test?

[OverLordGoldDragon]

Sure:

import pytest
import matplotlib.pyplot as plt

def test_plot():
    plt.plot(list(range(10)))
    plt.show()  # no stall if commented, but no plot either

if __name__ == '__main__':
    pytest.main([__file__, "-s"])

[StefRe]

spyder-unittest runs the tests as a regular python process (i.e. not on a ipython kernel). I'm not sure if it's possible to use spyder's inline backend (the plots pane) for rendering matplotlib output from such a process.

However, in my opinion, unit tests are supposed to run automatically without user interaction. If the plot is (part of) the test result then the test should assert that it equals the expected result. You can use compare_images for this:

import pytest
import matplotlib as mpl
import matplotlib.pyplot as plt
from matplotlib.testing.compare import compare_images

def test_plot_compare(tmp_path):
    expected = 'baseline_test_plot.png'
    actual = str(tmp_path / 'test_plot.png')
    with mpl.rc_context(rc=mpl.rc_params()):
        plt.plot(list(range(10)))
        plt.savefig(actual)
    
    assert compare_images(expected, actual, 0) is None

if __name__ == '__main__':
    pytest.main([__file__, "-s"])

As a bonus you also get the difference image to see how exactly the actual result differs from the expected.

The context manager is used here to make the test run correctly both on the spyder console and from the unittest plugin as the matplotlib rc settings used by spyder differ from the default values (figsize, dip, subplot.bottom). So if you create your baseline image in spyder and then run the test from the plugin it will fail unless you enforce the same parameters.

[OverLordGoldDragon]

@StefRe Thanks for the suggestion. From what I can tell, compare_images tests (1) correctness of figure saving, (2) Spyder vs. unittest rc settings. I never found (1) a problem, and designed all configs in Spyder to begin with, so both are redundant in my use. It further introduces much additional boilerplate testing code for testing a dedicated visualization package, easily at least doubling code length - calls are made to functions within functions within classes, with dict-customization, so digging in each to reproduce for compare_images would be vastly counterproductive. (Also much of figs never save, and some close / delete before returning from function call, making this workaround outright impossible).

[StefRe]

I guess we talk about different things, maybe I couldn't make myself clear or I misunderstood your question in the first place.
Compare_images does not test anything (figure saving or settings etc.) It's just an instrument to automatically compare the actual image produced by the test with an expected image (created manually before the test). It just replaces the visual verification you do when looking at the image by an automatice assertion. So I was under the impression that the image is the result of your code under test and that the test should assert that the correct image was generated. If this is not your intention then you could just prevent any plotting by matplotlib.use('template') (but you wrote in your question that "silencing isn't always an option").

[OverLordGoldDragon]

@StefRe By "silencing isn't always an option" I meant I need to personally validate generated plots (can't code "check if annotation floats nicely near histogram"). True that I don't always need to do this across all files at once, and that a workaround is to silence everything most of the time to get coverage statistics - but that's just ignoring the problem, not fixing it. Admittedly I need the plots to also be generated inline, as sorting through dozens of generated windows manually is a no-no - so this complicates things further (for Spyder).

Regardless, it's better than nothing. The issue can then be considered a "feature request". As far as a bug status, I'll call the issue resolved.

[jitseniesen]

I misunderstood the issue. I thought that the plugin behaves differently from running pytest directly (from the command line), which I would consider a bug. However, the problem is that it behaves differently from running in a Spyder console. That is fine with me, the Spyder console is meant to be used interactively and tests (normally) not.

If you need to do manual validation, it is not automatic testing, so this seems to be a rather atypical use case. I don't think I'll be working on it anytime soon, but I'm happy to keep the issue open.