Unitests: move environment setup from run_unitests.py to test fixtures

[dcbaker]

We currently have the run_unttests.py removing certain environment variables (CFLAGS and friends) before starting tests, because some tests do not behave correctly when they are set. The right place to do this is an a test fixture, particularly in setUpClass.

To facilitate this, a new base class has been added that adds this setup through a mock, and several test classes that should have this setup have been extended to use it. Then the environment setup is moved into the fixture.

Apart from being more correct, this allows running tests without the runner to be more useful, namely if one wants to invoke pytest or python -m unittest directly.

[eli-schwartz]

Apart from being more correct, this allows running tests without the runner to be more useful, namely if one wants to invoke pytest or python -m unittest directly.

I have some skepticism of this PR, mostly because I do not understand why you think you should be doing this.

[dcbaker]

Apart from being more correct, this allows running tests without the runner to be more useful, namely if one wants to invoke pytest or python -m unittest directly.

I have some skepticism of this PR, mostly because I do not understand why you think you should be doing this

Because I want to use standard tools in the standard way rather than through a project specific wrapper script.

run_unitests.py offers me 0 advantages over running pytest directly (except for this case of test environment setup being done in the wrapper instead of in a test fixture where it belongs, so it papers over a bug), and numerous disadvantages, among them:

• it automatically turns on -n auto, a behavior I don't want. If I invoke without -n I expect to get a serial test run.
• it mangles the -k option making it more difficult than it already is to write test filters
• When I press ctrl-c it gives piles of backtraces instead of just cleanly existing, a thing I am prone to do when every test starts failing.

Rather than trying to make this custom wrapper not terrible for me as a developer, I can just accept that it isn't meant for me to use as a developer, it's meant to make running our unittests in CI work well, a job it is doing just fine at.

[eli-schwartz]

Can you demonstrate for example how to implement --backend without run_unittests.py?

[dcbaker]

MESON_UNIT_TEST_BACKEND=xcode pytest

I'm honestly frustrated with the whole discussion, because it feels like we're blocking what is clearly an improvement in the form "use test fixtures to ensure that the environment matches the test expectations" on what feels to me like animosity toward me not wanting to use run_unittests.py

[eli-schwartz]

I just think it feels overengineered. :/

#13423 attempted the same thing, but by moving "test harness" setup to unittests/__init__.py and performing it a single time at import, which felt a bit more straightforward. Creating a setupClass that potentially runs setup and teardown on every single test, just for something that is supposed to run at program startup, feels like we're learning the wrong lesson here.

[dcbaker]

Well, I really wanted to move to pytest because pytest has global fixtures, not just test and class fixtures.

These are class fixtures, so they're only run once per class and not per test.

[eli-schwartz]

These are class fixtures, so they're only run once per class and not per test.

They may be run once per test, if tests from different classes are interleaved together. There are some plugins that do this automatically whenever they are installed, so you might not even realize that's what happened.

[jpakkane]

Well, I really wanted to move to pytest because pytest has global fixtures, not just test and class fixtures.

We can't really "move" to pytest because it is not in the stdlib. Plain unittest will still have to work so any functionality that is not in it has to be hand rolled anyway.

[bonzini]

Even if meson cannot require more than the stdlib to run, that would not exclude relying on pytest or other external Python-only wheels for the purpose of running tests, would it?

[eli-schwartz]

I've found it useful in the past to be able to run the self-tests on systems without pytest installed. In particular, running against additional python interpreter versions, where pytest was only packaged / installed for a single (default) python.

Requiring pytest would make it a hassle for me to do cpython alpha testing.

[dcbaker]

I understand the reasons why we're not going to, but it's pretty frustrating because unittest is... not great. As we see here.

[dcbaker]

Stack overflow has some suggestions on global setup/teardown for unittest, I'll have to try those in a bit.

[eli-schwartz]

We really don't see that here. :) A pytest session fixture is often overly complicated as opposed to just running code at startup. The point of fixtures is to tag them for use on specific tests (by passing them as magic test function arguments), which allows you to do things like have a fixture that runs only once per pytest run, but only right before the first function that uses that fixture, and avoid running the fixture if you don't run any tests that use the fixture (only tests that don't use it).

It's trivial to remap the current environment setup into an __init__.py, @lazka already submitted a PR to do so but never responded to my review comment there.

[dcbaker]

What I mean is, there's a mechanism unittest provides for a global setup/teardown you put in your __init__.py, but is guaranteed to work the way you expect.