support running and debugging pytest for local tree

[thorsten-klein]

Why?

I'm not sure how tests for the local source tree are currently debugged.
In my setup, I’m unable to run pytest directly on the local tree, and debugging is difficult because it’s not possible to step through since subprocess is used in the current test setup.

Proposed Changes

This change allows running pytest directly, instead of only being able to run it via uv run poe test.
pytest can test either the installed West package or the local source tree, depending on how it’s invoked.
This simplifies test execution and improves compatibility with pytest integrations (e.g. in IDEs).

With this setup, pytest is fully in charge for setting up the Python environment, so testing becomes straightforward:

# Test the installed West package
pytest

# Test the local source tree
pytest -o pythonpath=src
# or
PYTHONPATH=src pytest

Background

During this work, I faced that many tests used subprocess, which interfered with testing the local copy.
Subprocesses do not inherit the full Python environment configured by pytest, causing a mix between modules from the installed West package and those from the local source tree. Therefore I removed the use of subprocesses in tests wherever possible. Where subprocesses are still used, the python module (west/app/main.py) is called instead of west whereby it is ensured now that its correct modules are used by prepending correct PYTHONPATH. With those changes only one Python environment managed by pytest is used, ensuring consistent behavior and enabling proper debugging.

[pdgendt]

I'm not sure I like where this is going. Testing west should focus on the package. I think we could add some documentation bits for people who really, really want to use pytest directly that they should install an editable version with

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install -e .

[thorsten-klein]

Note: A few other tests invoke the hardcoded west executable directly, rather than using cmd().
As a result, they only work if west is installed and available in PATH.

For consistency and robustness, these tests should be refactored so that pytest passes even when west is not installed.
Would you like me to address this in the current PR, or handle it in a follow-up PR?

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.32%. Comparing base (a53dbf4) to head (e24ae8e).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #859      +/-   ##
==========================================
- Coverage   84.34%   84.32%   -0.02%     
==========================================
  Files          11       11              
  Lines        3366     3369       +3     
==========================================
+ Hits         2839     2841       +2     
- Misses        527      528       +1     

Files with missing lines	Coverage Δ
src/west/app/main.py	75.91% <100.00%> (+0.31%)	⬆️

... and 1 file with indirect coverage changes

[pdgendt]

I recommend you check https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code too.

[thorsten-klein]

I'm not sure I like where this is going. Testing west should focus on the package. I think we could add some documentation bits for people who really, really want to use pytest directly that they should install an editable version with

Installation in editable mode does not work for me:

ERROR: Project file:///home/user/work/GIT/west has a 'pyproject.toml' and
its build backend is missing the 'build_editable' hook. Since it does not have
a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode.
Consider using a build backend that supports PEP 660.

Am I doing something wrong?

I recommend you check https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code too.

In this documentation it says:

If you don’t use an editable install and are relying on the fact that Python by default 
puts the current directory in sys.path to import your package, you can execute 
python -m pytest to execute the tests against the local copy directly, without using pip.

So this is exactly what I want to do. I want to run python -m pytest and execute the tests against the local copy.
When the tests are run in tox, then they run against the installed package.
Therefore I adapted the tests to prefer the installed package (if west in correct version exists), and otherwise fall back to the local copy (main.py).

In many other Python open-source projects, it’s possible to run pytest directly, and I find that very convenient.

if you ran pip install -e . before, the installed west executable will probably be used, while not desired.

If the installed west version matches, then the tests should test against this installed package.
So when installed with pip install -e . the west version should always be up-to-date.

The main problem seems to lay in current tests that call subprocess, which does not inherit the sys.path from the test environment. Therefore pytest does not work as expected, since different west modules are imported in the test versus the called west process.

Running pytest directly becomes particularly useful when debugging in an IDE, where for example some paths from ~/.bashrc are not set when IDE is started via GUI.
Currently, I have to either start my IDE inside a prepared virtual environment or I have to install the tool I want to test into my IDE environment. That feels wrong.

---

Out of curiosity: Have you ever considered distinguishing between different test levels and handling them different (e.g., unit tests vs. integration tests)? (Does not matter)

[pdgendt]

Installation in editable mode does not work for me:

ERROR: Project file:///home/user/work/GIT/west has a 'pyproject.toml' and
its build backend is missing the 'build_editable' hook. Since it does not have
a 'setup.py' nor a 'setup.cfg', it cannot be installed in editable mode.
Consider using a build backend that supports PEP 660.

Am I doing something wrong?

So what I do to test (requires an environment!):

$ python3.10 -m venv .venv
$ source .venv/bin/activate
$ pip install -e .

In many other Python open-source projects, it’s possible to run pytest directly, and I find that very convenient.

I get that, but I also want to make sure developers are running the version they're expecting to run. Maybe we should check POE_ACTIVE and in that case require an installed version.

Running pytest directly becomes particularly useful when debugging in an IDE, where for example some paths from ~/.bashrc are not set when IDE is started via GUI. Currently, I have to either start my IDE inside a prepared virtual environment or I have to install the tool I want to test into my IDE environment. That feels wrong.

Right, but I'd like to avoid having to mess with the system path if possible. There should be some recommended practices for this, no? There are so many packages out there, we can't be the only with this problem.

I guess that's why the myriad of task runners exist.

[thorsten-klein]

Right, but I'd like to avoid having to mess with the system path if possible. There should be some recommended practices for this, no? There are so many packages out there, we can't be the only with this problem.

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()?
This is at least how conan seems to do it:
Ref:

• https://github.com/conan-io/conan/blob/develop2/test/integration/command/test_build.py (using TestClient.run() which is analog to west's cmd())
• https://github.com/conan-io/conan/blob/develop2/conan/test/utils/tools.py#L569 (implementation of TestClient.run calls _run_cli() )
• https://github.com/conan-io/conan/blob/develop2/conan/test/utils/tools.py#L553 (implementation of the _run_cli method without any subprocess)

With this, pytest can completely ensure which modules are tested, because there are no other processes that have different PYTHONPATH.
This is also the benefit of the src layout, as described in your mentioned link (https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code)

pytest # test installed package
PYTHONPATH=src pytest # test local package

The benefit of this would also be that the tests become fully debuggable.
Currently. subprocess cannot be fully debugged, since the whole west logic happens in this other process, which is only one python call.

[pdgendt]

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()? This is at least how conan seems to do it: Ref:

Yes please, but I guess that this is even more work than the current proposal?

With this, pytest can completely ensure which modules are tested, because there are no other processes that have different PYTHONPATH. This is also the benefit of the src layout, as described in your mentioned link (docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code)

pytest # test installed package
PYTHONPATH=src pytest # test local package

The benefit of this would also be that the tests become fully debuggable. Currently. subprocess cannot be fully debugged, since the whole west logic happens in this other process, which is only one python call.

Getting rid of the subprocess would be nice, we can also get rid of the patch for coverage.

[thorsten-klein]

I will give it a try 👍

[mbolivar]

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()?

Just FYI that's technically an abstraction violation; nothing in west.app is API

[pdgendt]

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()?

Just FYI that's technically an abstraction violation; nothing in west.app is API

But that doesn't mean we can't rely on it for testing, right? From a practical point of view it would solve some of the issues we have with testing/coverage.

[mbolivar]

No, it's just white-box testing

[thorsten-klein]

I use main.main() now for testing, which takes argv as argument. This argv is the west command line API which should be stable, so it can be used for testing 😇

[marc-hb]

These changes allow running pytest directly, simplifying test execution and making debugging even easier.

Shorter commands are always nicer but it's IMHO not enough to justify a PR that big. Is there some bigger problem that this PR solves for you? I remember I used to struggle to use pudb with tox but now this seems enough for me:

uv run pytest -k narrow -s

[marc-hb]

Is there some bigger problem that this PR solves for you?

I scanned the comments and found a couple problem descriptions that should have been in commit messages. And then some more:

Wouldn't it make sense to get rid of this subprocess completely and instead use WestApp.run()?

Yes please, but I guess that this is even more work than the current proposal?

Prototyping and experimenting is great, but once a PR leaves the draft status it should be clear 1. what is the problem solved 2. why and how that solution was chosen.

For earlier stage discussions please use drafts, issues, discord,...

Some good inspiration:
https://github.com/zephyrproject-rtos/zephyr/blob/67435e394f0d025b8/.github/ISSUE_TEMPLATE/002_enhancement.yml

[thorsten-klein]

1. what problem does this solve

As a developer, I want to easily run the project’s tests using pytest directly in my local environment.
Although the README.md mentions that tests can be executed with pytest, this currently doesn’t work because certain prerequisites must be met (for example, west must be installed into the environment).
I understand that this setup may be a leftover from earlier times when tox was used, but now it should be possible to run pytest on the local copy without having to install the project I want to test.

This is matching to what pytest proposes under "Benefits".
Ref: https://docs.pytest.org/en/7.1.x/explanation/goodpractices.html#tests-outside-application-code

Debugging is also challenging under the current approach since many tests rely on subprocess calls.
Those subprocesses run in a separate environment (potentially even with different module versions), which makes it difficult to inspect variables or step through the code during debugging.

I’m not sure how you currently debug tests in your IDE, but most IDEs offer native pytest integration — something that doesn’t work well with the current test structure.

2. why and how this solution was chosen

The proposed change minimizes or eliminates the use of subprocess in tests, allowing them to run directly in the current Python process — which makes debugging far easier.

Additionally, it introduces flexibility by allowing developers to choose what is tested using standard pytest options:

# test the installed west 
pytest

# test the local copy 
pytest -o pythonpath=src
PYTHONPATH=src pytest

---

I hope these arguments are convincing.

PS: Up to now, I’ve been developing west using only print statements for debugging — which has been quite painful.
Often I had to use print('test', file=sys.stderr) so that I can see my output at all within tests, because of the redirected subprocess stdout. I’ve never been able to get proper debugging working in my IDE.

That’s why I really, really hope you’ll accept this Pull Request — it is state-of-the-art and it will make development and debugging so much easier.

[pdgendt]

Could you cleanup your commits a bit, please?

• Use the body to describe the change and its rationale
• Don't add stuff that is removed in the next commit (tool.pytest.ini_options in pyproject.toml)
• Split test commits from other changes

[marc-hb]

I’m not sure how you currently debug tests in your IDE, but most IDEs offer native pytest integration — something that doesn’t work well with the current test structure.

Thanks, I guess this is how this PR should have all started. Please write a more detailed commit message based on that and start a fresh, clean PR (referring to this one). So far this PR looks more like brainstorming. Brainstorming is great! It's just confusing when it happens in a non-draft PR.

Could you cleanup your commits a bit, please?

See above :-)

Debugging is also challenging under the current approach since many tests rely on subprocess calls.
Those subprocesses run in a separate environment (potentially even with different module versions), which makes it difficult to inspect variables or step through the code during debugging.

I suspect this is a different issue that requires a different solution. If this can be solved at a different time, then it should. Should it? Clearly state which way each PR is going.

FYI I've been using PUDB_TTY to avoid that problem
https://documen.tician.de/pudb/starting.html#debugging-from-a-separate-terminal

[marc-hb]

I’m not sure how you currently debug tests in your IDE, but most IDEs offer native pytest integration — something that doesn’t work well with the current test structure.

It's not "pytest integration" and you may not even call this an "IDE" but FWIW this works fine with Emacs right now:

Esc-x pdb
uv run pytest -k narrow --trace

It works even remotely over ssh/Tramp.

However this is only debugging the pytest code, NOT the actual west code for the subprocess reasons already discussed above.

[pdgendt]

I'm having some trouble with getting this to work locally. My assumption was that I should now be able to do:

$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install -e .
$ pip install pytest
$ pytest

However this fail, for example with a single test: (same with -o pythonpath=src)

$ pytest tests/test_help.py::test_extension_help_and_dash_h
============================= test session starts ==============================
platform linux -- Python 3.12.11, pytest-8.4.2, pluggy-1.6.0
rootdir: /home/pdgendt/west
configfile: pyproject.toml
collected 1 item

tests/test_help.py F                                                     [100%]

=================================== FAILURES ===================================
________________________ test_extension_help_and_dash_h ________________________

west_init_tmpdir = local('/tmp/pytest-of-pdgendt/pytest-9/test_extension_help_and_dash_h0/workspace')

    def test_extension_help_and_dash_h(west_init_tmpdir):
        # Test "west help <command>" and "west <command> -h" for extension
        # commands (west_init_tmpdir has a command with one).
    
>       cmd('update')

/home/pdgendt/west/tests/test_help.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/home/pdgendt/west/tests/conftest.py:401: in cmd
    _cmd(cmd, cwd, env)
/home/pdgendt/west/tests/conftest.py:386: in _cmd
    raise e
/home/pdgendt/west/tests/conftest.py:383: in _cmd
    main.main(cmd)
/home/pdgendt/west/src/west/app/main.py:1205: in main
    app.run(argv or sys.argv[1:])
/home/pdgendt/west/src/west/app/main.py:284: in run
    self.run_command(argv, early_args)
/home/pdgendt/west/src/west/app/main.py:588: in run_command
    self.run_builtin(args, unknown)
/home/pdgendt/west/src/west/app/main.py:702: in run_builtin
    self.cmd.run(args, unknown, self.topdir,
/home/pdgendt/west/src/west/commands.py:201: in run
    self.do_run(args, unknown)
/home/pdgendt/west/src/west/app/project.py:1208: in do_run
    self.update_all()
/home/pdgendt/west/src/west/app/project.py:1274: in update_all
    self.update(project)
/home/pdgendt/west/src/west/app/project.py:1522: in update
    self.update_submodules(project)
/home/pdgendt/west/src/west/app/project.py:1423: in update_submodules
    self.die(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <west.app.project.Update object at 0x7a3b414c5e50>, exit_code = 1
args = ('Submodule status failed for project: Kconfiglib.',)

    def die(self, *args, exit_code: int = 1) -> NoReturn:
        '''Print a fatal error using err(), and abort the program.
    
        :param args: sequence of arguments to print.
        :param exit_code: return code the program should use when aborting.
    
        Equivalent to ``die(*args, fatal=True)``, followed by an attempt to
        abort with the given *exit_code*.'''
        self.err(*args, fatal=True)
        if self.verbosity >= Verbosity.DBG_EXTREME:
            raise RuntimeError("die with -vvv or more shows a stack trace. "
                               "exit_code argument is ignored.")
        else:
>           sys.exit(exit_code)
E           SystemExit: 1

/home/pdgendt/west/src/west/commands.py:525: SystemExit
---------------------------- Captured stdout setup -----------------------------
initializing session repositories in /tmp/pytest-of-pdgendt/pytest-9/session_repos0
Initialized empty Git repository in /tmp/pytest-of-pdgendt/pytest-9/session_repos0/Kconfiglib/.git/
[master (root-commit) 58e308a] initial c41dc662-4a8a-4334-9138-b53799fdadce
Initialized empty Git repository in /tmp/pytest-of-pdgendt/pytest-9/session_repos0/tagged_repo/.git/
[master (root-commit) e97846e] initial d762cb97-fd56-4abe-a5cb-1db47d829573
Initialized empty Git repository in /tmp/pytest-of-pdgendt/pytest-9/session_repos0/net-tools/.git/
[master (root-commit) 78efb82] initial baeb89cf-ef96-4f14-890d-525a290af83e
Initialized empty Git repository in /tmp/pytest-of-pdgendt/pytest-9/session_repos0/zephyr/.git/
[master (root-commit) 5368f34] initial f7d2e202-c0a3-435a-8bbe-cb6ac04b1adc
[master f340070] base zephyr commit
 3 files changed, 2 insertions(+)
 create mode 100644 CODEOWNERS
 create mode 100644 include/header.h
 create mode 100644 subsys/bluetooth/code.c
[zephyr 7e0c488] test kconfiglib commit
 1 file changed, 1 insertion(+)
 create mode 100644 kconfiglib.py
[master ef483c3] tagged_repo commit
 1 file changed, 1 insertion(+)
 create mode 100644 test.txt
[master 3375f40] test net-tools commit
 3 files changed, 18 insertions(+)
 create mode 100644 qemu-script.sh
 create mode 100644 scripts/test.py
 create mode 100644 scripts/west-commands.yml
finished initializing session repositories
[master 0f868d9] add manifest
 1 file changed, 26 insertions(+)
 create mode 100644 west.yml
---------------------------- Captured stderr setup -----------------------------
Switched to branch 'zephyr'
Cloning into 'Kconfiglib'...
done.
Cloning into 'tagged_repo'...
done.
Cloning into 'net-tools'...
done.
Cloning into 'zephyr'...
done.
Cloning into '/tmp/pytest-of-pdgendt/pytest-9/test_extension_help_and_dash_h0/workspace/.west/manifest-tmp'...
done.
----------------------------- Captured stderr call -----------------------------
Cloning into bare repository '/home/pdgendt/.cache/zephyrproject/Kconfiglib/0aa789fdcd0cd5b2a14b654f054b65ae'...
done.
Cloning into '/tmp/pytest-of-pdgendt/pytest-9/test_extension_help_and_dash_h0/workspace/subdir/Kconfiglib'...
done.
From /home/pdgendt/.cache/zephyrproject/Kconfiglib/0aa789fdcd0cd5b2a14b654f054b65ae
 * branch            zephyr     -> FETCH_HEAD
HEAD is now at 7e0c488 test kconfiglib commit
FATAL ERROR: Submodule status failed for project: Kconfiglib.
=========================== short test summary info ============================
FAILED tests/test_help.py::test_extension_help_and_dash_h - SystemExit: 1
============================== 1 failed in 0.24s ===============================

Running with uv also results in the same errors.

EDIT: I actually have this on main 🤔 need to look into this.

[pdgendt]

Running with uv also results in the same errors.

EDIT: I actually have this on main 🤔 need to look into this.

@thorsten-klein see #862

[Copilot]

Pull Request Overview

Enable running and debugging pytest directly against the local source tree without relying on subprocess-based invocations, improving IDE integration and consistency of environment handling.

• Replace subprocess-based test helpers with in-process CLI invocation (west.app.main), adding a subprocess variant only where needed
• Update tests to expect SystemExit, capture stderr explicitly, and add coverage for module execution and forall env vars
• Adjust main entrypoint to prepend src to sys.path when run as a script/module; update coverage paths and README instructions

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
tests/test_project.py	Replace CalledProcessError with SystemExit, switch to cmd_subprocess where necessary, add test_forall_env_vars, and separate stderr assertions.
tests/test_main.py	Add tests for version output via in-process and subprocess calls; add module-run test verifying sys.path injection and exit code.
tests/test_help.py	Simplify help output checks by removing platform newline normalization (now captured consistently).
tests/test_config.py	Migrate exception expectations from CalledProcessError to SystemExit and update cmd_raises usage.
tests/test_alias.py	Update exception expectations, but assertions inspect exit code instead of error output.
tests/conftest.py	Introduce _cmd, cmd, cmd_raises, cmd_subprocess helpers; capture stdout/stderr; run west.app.main directly; optional subprocess mode.
src/west/app/main.py	Prepend src to sys.path when executed as main to ensure local modules are used.
pyproject.toml	Fix coverage omit globs to match paths nested under any directory.
README.rst	Document how to run pytest against installed package vs local source using pythonpath=src.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}