RFE: adapt exsting test suite to pytest

[kloczek]

Describe the bug
Looks like pyetst is failing because it cannot find TestCmd module.

Required information

• Version of SCons: 4.5.
• Version of Python: 3.8.16
• How you installed SCons: N/A
• What Platform are you on? Linux x86/64
• How to reproduce your issue?

• python3 -sBm build -w --no-isolation
• because I'm calling build with --no-isolation I'm using during all processes only locally installed modules
• install .whl file in </install/prefix>
• run pytest with $PYTHONPATH pointing to sitearch and sitelib inside </install/prefix>
• build is performed in env which is cut off from access to the public network (pytest is executed with -m "not network")

• How you invoke scons (The command line you're using "scons --flags some_arguments")
  N/A

Here is pytest output:

+ PYTHONPATH=/home/tkloczko/rpmbuild/BUILDROOT/python-scons-4.5.1-2.fc35.x86_64/usr/lib64/python3.8/site-packages:/home/tkloczko/rpmbuild/BUILDROOT/python-scons-4.5.1-2.fc35.x86_64/usr/lib/python3.8/site-packages
+ /usr/bin/pytest -ra -m 'not network'
============================= test session starts ==============================
platform linux -- Python 3.8.16, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/tkloczko/rpmbuild/BUILD/scons-4.5.1
collected 0 items / 5 errors

==================================== ERRORS ====================================
______________________ ERROR collecting src/test_files.py ______________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_files.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/test_files.py:38: in <module>
    import TestSCons
E   ModuleNotFoundError: No module named 'TestSCons'
___________________ ERROR collecting src/test_interrupts.py ____________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_interrupts.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/test_interrupts.py:36: in <module>
    import TestSCons
E   ModuleNotFoundError: No module named 'TestSCons'
____________________ ERROR collecting src/test_pychecker.py ____________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_pychecker.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/test_pychecker.py:33: in <module>
    import TestSCons
E   ModuleNotFoundError: No module named 'TestSCons'
______________________ ERROR collecting src/test_setup.py ______________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_setup.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/test_setup.py:41: in <module>
    import TestSCons
E   ModuleNotFoundError: No module named 'TestSCons'
_____________________ ERROR collecting src/test_strings.py _____________________
ImportError while importing test module '/home/tkloczko/rpmbuild/BUILD/scons-4.5.1/src/test_strings.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib64/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
src/test_strings.py:39: in <module>
    import TestCmd
E   ModuleNotFoundError: No module named 'TestCmd'
=========================== short test summary info ============================
ERROR src/test_files.py
ERROR src/test_interrupts.py
ERROR src/test_pychecker.py
ERROR src/test_setup.py
ERROR src/test_strings.py
!!!!!!!!!!!!!!!!!!! Interrupted: 5 errors during collection !!!!!!!!!!!!!!!!!!!!
============================== 5 errors in 1.11s ===============================

I see that module in the tree however look like it has location which does not allow use pytest

[tkloczko@pers-jacek scons-4.5.1]$ find . -name TestCmd\*
./testing/framework/TestCmd.py
./testing/framework/TestCmdTests.py

It woild be nice to have possibility to test scons using pytest.

[bdbaddog]

This is not a bug, but a feature request.
SCons is meant to be tested by it's own framework.

If you'd like to submit a PR to enable using pytest.

[mwichmann]

On March 7, 2023 4:57:44 AM MST, "Tomasz Kłoczko" ***@***.***> wrote: **Describe the bug** Looks like pyetst is failing because it cannot find `TestCmd` module.

Not quite sure what you're after here. Yes, using pytest would be nice, but SCons has its own testing framework that extends unittest, and a very long history of that, it would be kind of hard to disentangle. You need to look at the custom test runner, cleverly named `runtest.py`. It stuffs the path to the running SCons and the path to the framework (`testing/framework`) into `PYTHONPATH`, along with some other environment fiddling (things the framework is going to look at). Also note scons unit tests don't follow the pytest naming convention, all the test files end in Tests.py (including the capital T). runtest has its own discovery mechanism.

[kloczek]

Adaptation to pytest wil involve main test sulte subtree reorganisation,
It will require as well relative deep knowledge about existing test suite.

Nevertheless pytest is way more advanced tan unittest and I woild really encourage to move to pytest as it has tons of extensions which simplifies testing. Only from that point of view it may be worth to start thing about that by simple ctreate standard tests directory and start copy bit by bit test suite to that subtree.
With such approach until would be still possible to use existing test suite.

[bdbaddog]

Honestly we're unlikely to get to this anytime soon.
Our current test framework works well for our project needs.

We have many other priorities and limited resources.

As always, if you're up for taking this project on, please do.
But, that said, whatever the update it would have to be a drop in replacement and not require overhauling too much of the existing tests. Because then we'd need to revalidate all those tests as well as the new test framework.

[mwichmann]

Don't disagree about the benefits of pytest, I've looked at this too. Have recently started introducing some of the specialized unittest asserts (test.assertEqual, etc.) because they provide more information without handcrafting the failure messages - but pytest has the smarts to do the same with just plain assert statements, i.e. you get details "for free".

As @bdbaddog says, this isn't likely to be any kind of high priority at this time. And the organization is relatively intentional - the original developers wanted to keep code/test/doc in proximity, so there is often a pattern of (SCons/Module.py, SCons/ModuleTests.py, SCons/Module.xml), rather than the current Python project recommendation of having those each in their own directory tree. It might take some convincing to move away from that, though nothing is impossible...

[kloczek]

Whatabout diffrent approach by use just pytest temporary to call existing test sute? 🤔

This woild allow to reuse 100% existing test suite and add completly new usinits only in pytest ..

[bdbaddog]

Let me reiterate:
"We have many other priorities and limited resources."
This is not really anywhere in the priority list currently.

[mwichmann]

For grins, I ran the existing unittests (which are the minority of tests in the entire thing we call the test suite. It was relatively easy to do: you need to pass testing/framework in PYTHONPATH, and you need to change the test discovery since we name unittest files differently. An entry in pytest.ini was enough to do that:

[pytest]
python_files = *Tests.py

The results were: the tests ran, but ran fairly poorly. Had hoped for maybe a bit better, since pytest knows how to run unittest-based tests. It seems pytest is a lot fussier about scope, and so many things that are imported or declared in the module global scope and work in unittest are not visible in parts of the testing under pytest.

====== 594 failed, 347 passed, 8 skipped, 3 warnings, 15 errors in 15.85s ======