Test suite improvements

[Flamefire]

Extracted some parts of #3780

Major change: Running the suite now allows to pass any valid unittest args, such as -k for filtering or --verbose for more detailed output (really missed that in #3780) similar to easybuilders/easybuild-easyconfigs@17379db in the EC repo

• Requires Use skip markers in tests #4188
• Requires silence distutils deprecation warnings #4204

[Flamefire]

I remember something about 9d04087
The "proper" way just marks the test as skipped and hence is no longer caught by the filtered output check. I.e. previously if a test was skipped it would print something to stdout which the later check would find and fail CI. This no longer works (this way).

So maybe the check should look for skipped tests? Or just ignore this for now...

[boegel]

These changes break the way we're currently filtering tests from test.framework.suite, which we should try and avoid:

$ python3 -m test.framework.suite load
E
======================================================================
ERROR: load (unittest.loader._FailedTest)
----------------------------------------------------------------------
AttributeError: module '__main__' has no attribute 'load'

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (errors=1)

This is supposed to filter all tests down to just the ones that include load in the test name...

[Flamefire]

These changes break the way we're currently filtering tests from test.framework.suite, which we should try and avoid:

But they allow the proper way: python3 -m test.framework.suite -k load should work which it did not before especially for the non-standard way of adding subtests. See e.g. 4e0a269

Edit: BTW: The failing test is due to a faulty usage env_file = open(reprod_dumpenv, "r").read().
Replace this by read_file to avoid the resource leak (missing close for open)

[boegel]

These changes break the way we're currently filtering tests from test.framework.suite, which we should try and avoid:

But they allow the proper way: python3 -m test.framework.suite -k load should work which it did not before especially for the non-standard way of adding subtests. See e.g. 4e0a269

OK, but can we somehow automagically translate python3 -m test.framework.suite load to python3 -m test.framework.suite -k load, to ensure both work?

Many people are used to filtering tests by just passing a pattern, and this still works for the individual subsuites:

python3 -m test.framework.modules load

(so does -k load here)

[Flamefire]

OK, but can we somehow automagically translate python3 -m test.framework.suite load to python3 -m test.framework.suite -k load, to ensure both work?

Not sure. I'm afraid that would be hacky and could be ambiguous and/or break things. As far as I can tell this way now is the "correct" way (i.e. as documented by unittest) so if people now have to pass an additional -k I guess they'll manage ;) How many are running the EB test suite anyway? And if we have to decide between having access to (all) the regular unittest flags or this little backwards compat thing I'd say the change is well worth it

[Flamefire]

Rebased this and the other PR. I just ran again in a case where I would have wanted to use e.g. -v

I found a way to support the current test filtering: If the 2nd argument starts with a dash the new method is used, otherwise the old one. So when using unittest you can do python -m test.framework.suite -v -k '*robot*' and also python -m test.framework.suite robot to use the "old" filtering only