fix: Allow west commands to be imported from a project subdirectory if manifest is located in subdirectory

[nmunnich]

Fixes #725

Note that I relied on the test to tell me if the fix is working, I did not test it with a proper repository setup. Let me know if this is desired/this project is one where manual testing of such kind is necessary.

I'm unfamiliar with west release cycles - if this is accepted, could someone let me know when I could expect to see a new version of west released, so that we can start recommending setups relying on this to our users?

[marc-hb]

Thank you so much for taking this! See comments below.

Sorry if some comments appear twice, I had to make them again because you force pushed in the mean time. Please keep force-pushing, it's just Github support for force-pushes is pretty bad.

[nmunnich]

Thanks for the quick review! (I hope the linter is happy with me this time, uv run poe all doesn't seem to check whatever that last test is)

[nmunnich]

Ah, the formatter does run, but for whatever reason it thinks it needs to format 22 files (with no difference made to them) on my machine. I've kept that formatting on manifest.py, maybe now it'll be happy? Just another quirk of Windows I suppose.

[codecov]

Codecov Report

❌ Patch coverage is 88.00000% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 86.09%. Comparing base (aed10a6) to head (e4d0866).
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/west/manifest.py	85.00%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #920      +/-   ##
==========================================
- Coverage   86.09%   86.09%   -0.01%     
==========================================
  Files          11       11              
  Lines        3474     3488      +14     
==========================================
+ Hits         2991     3003      +12     
- Misses        483      485       +2     

Files with missing lines	Coverage Δ
src/west/commands.py	95.63% <100.00%> (+0.03%)	⬆️
src/west/manifest.py	95.41% <85.00%> (-0.15%)	⬇️

[nmunnich]

Hmmm. I'm not sold on adding tests to test that the exceptions are indeed raised. Don't think the AssertionError can ever be raised anyway, it's just good practice to have it there.

[marc-hb]

Hmmm. I'm not sold on adding tests to test that the exceptions are indeed raised. Don't think the AssertionError can ever be raised anyway, it's just good practice to have it there.

I agree. Error-handling is notoriously lacking coverage across the entire industry (one of the reasons software is so insecure), but I don't think we need test coverage for raise InternalBug("this should never happen").

@pdgendt is big on test coverage and rightly so, let's wait until next week when he is back. Every PR needs 2 approvals anyway.

[Copilot]

Pull request overview

This pull request fixes issue #725 where imported west-commands from project subdirectories were not resolving correctly. When a manifest file is located in a subdirectory of a project (e.g., app/west.yml), and that manifest defines custom west-commands, the paths to those commands need to be resolved relative to the manifest's location, not the project root.

Changes:

• Modified _import_path_from_project to pass the manifest path instead of None to _import_data_from_project
• Updated _import_data_from_project to accept both _import_map and str types using match/case syntax, extracting the manifest directory and prepending it to west_commands paths
• Added a test case to verify that west-commands are correctly resolved when imported from project subdirectories

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
src/west/manifest.py	Updated import logic to correctly resolve west-commands paths relative to manifest subdirectory location
tests/test_manifest.py	Added test case to verify west-commands resolution from project subdirectories

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Path should be PurePosixPath to ensure cross-platform compatibility. Git stores paths internally using POSIX-style forward slashes, even on Windows. Using Path on Windows will create paths with backslashes, which may not work correctly. The codebase already uses PurePosixPath in similar contexts (see line 312).

Suggested change

	mfst_dir = Path(mfst_path).parent
	mfst_dir = PurePosixPath(mfst_path).parent

[pdgendt]

@marc-hb maybe related to #921 ?

[marc-hb]

Yes, related to #921 (which I just demoted to draft).

On one hand we should stick to Pathlib abstractions, avoid hardcoding forward slashes and not use Unix paths objects internally when running on Windows. This is not just about backslashes, there are other differences.

On the other hand we don't want west manifest --resolve to convert forward slashes to backslashes on Windows. EDIT: or do we? See #928

I need again more time to think about it and do more testing, sorry :-(

[marc-hb]

This is a much bigger can of worms than I expected. I wish I could "unsee" it but I can't :-(

For instance, on Windows:

west list -f '{path}'      # forward slashes
west list -f '{abspath}'   # backslashes 
west list -f '{posixpath}' # forward slash
west list topdir           # absolute and forward since #375

https://docs.zephyrproject.org/latest/develop/west/west-apis.html#west.manifest.Project

Also relevant:

• Overhaul path handling #411 and more
• topdir: west topdir prints posix style path. #375

[nmunnich]

Personally, I would argue that resolving that inconsistency is out of scope here, and I would prefer not to block waiting for a fix/resolution. Would you be alright with taking the PurePosixPath suggestion and going forward with that here?

[nmunnich]

I've gone through the pain of doing a bunch of testing of actual imports in PowerShell. This may be simpler than we think:

• No matter what path is set to, it always resolves in west manifest --resolve with /. So path: modules/\//\/\/\\\\/\//\./\zmk\/\/\/\.\\\/\test turned into modules/zmk/test.
• No matter what west-commands is set to, it remained the same in the west manifest. In addition, regardless of how many extra \ and / combinations I put, my dummy west command was being picked up and executed correctly.

I've now modified the code somewhat, so that the west-commands will always resolve with posix paths, for consistency with path. I noticed a comment on L2468 that seems relevant to the conversation, and took the approach used there.

I did notice another bug while doing my testing:

manifest:
  projects:
    - name: p1
      url: placeholder
      revision: west-test
      import: app/west.yml
      west-commands: app/scripts/west-commands.yml

west-commands:
  - file: test/west-commands/west.py
    commands:
      - name: west
        class: West
        help: Print hello world to the console

When resolving this, while the west-commands.yml file is located correctly and the commands are visible in west help, trying to actually execute the command doesn't work because west is trying to resolve it as <module-path>/test/west-commands/west.py rather than <module-path>/app/test/west-commands/west.py. My current thinking is that we need to add manifestdir as a property to Project, so that we can refer to it on L691 of commands.py. That feels like a rather extensive change though, would you have any alternative idea?

[marc-hb]

I've been working on adding some 2 "backslash" tests related to this (#928). I have this one ready but I want to finish a similar one for west-commands:
https://github.com/marc-hb/west/commits/backslash-tests/

I should be able to get the last one ready on Friday. I would like these "pure" tests to be merged before any change (feature or bug fix) in west itself.

Just a heads-up, sorry for not answering your comment yet - will do that on Friday too.

[marc-hb]

I would like these "pure" tests to be merged before any change (feature or bug fix) in west itself.

• Add test coverage for special characters in manifest and west commands #928

[marc-hb]

Sorry I'm paying attention to this only now:

I did notice another bug while doing my testing:

projects:
    - name: p1
      import: app/west.yml
      west-commands: app/scripts/west-commands.yml

First, I'm really not a fan of such "shortcuts + import" combinations BTW. This is adding west-commands from another git repo directly and "shortcutting" the import app/west.yml which could also add the same west-commands.yml file a second time! Does it get added a second time in your test case? I don't think you said. If yes, is that second time an error? A warning? I hope it's documented somewhere but I'm not holding my breath. Also, why adding the same west-commands.yml file twice?

If app/west.yml is not adding west-commands.yml second time then why is it not in app/west.yml? Why force every app/west.yml importer to manually add extensions?

Don't get me wrong: I don't see anything wrong with having the west-commands: shortcut ALONE without the import. But if a project becomes big and bold enough to have its own west.yml manifest, then shouldn't that submanifest import be a the single module "entry point" and take care of implementation details like the location of west-commands too? Or am I missing some real-world use case here?

All this being said, the ship has sailed: this has been allowed so the behavior should be specified. But: can we deal with this separately from #920? (which has got surprisingly more complicated than I expected already). If yes separately, could you please file a separate bug? Thx in advance.

cc:

• Clean way to import a west.yml's west commands but no submodule #903

[nmunnich]

In this case the bug I was trying to describe is that which is shown by the newest test and fixed in this PR in the last commit. Writing it this way was simply a way to avoid writing the app/west.yml where the script gets imported down, I think but can't recall completely that I simply didn't import the commands in app/west.yml when testing this.

The interaction I care about is that of the newest test, and I agree that any interaction with multiple import locations of the same westcommands file should be delt with separately from this.

backslashes :-(

[nmunnich]

Just an fyi, I think it is unlikely that I'll find the time to continue on this until next month. If it's still open in April I'll have another go, but I'm encouraging anyone who has the time to continue this sooner.

[marc-hb]

One more thought: this needs to be split into at least 2 commits, maybe 2 PRs:

1. the west-commands Path normalization
2. the prefix bug fix

I can do 1. after #928 is merged. I can probably rebase 2. as well, should be trivial enough (famous last words)

[nmunnich]

Rebased onto main. I also added a test and a fix for the west-commands not being imported correctly after the manifests are imported.

[marc-hb]

This PR opened a big backslash / PosixPath can of worms that kept both of us busy for a while in this comment: #920 (comment)

But I don't see any corresponding test as of the latest version 6f600e3.

Please:

• use special characters like I did in #928
• add some naive assert(west manifest --resolve) at the end of the relevant tests.

If this test is "naive" then I'm afraid it will fail either on Linux or on Windows but cannot pass on both at the same time. Then we can discuss whether we make the test Windows-specific like I did in #928, or whether we change west behavior to make the output the same on both. I will work on either option.

[nmunnich]

This PR opened a big backslash / PosixPath can of worms that kept both of us busy for a while in this comment: #920 (comment)

But I don't see any corresponding test as of the latest version 6f600e3.

Please:

* use special characters like I did in #928

* add some naive `assert(west manifest --resolve)` at the end of the relevant tests.

If this test is "naive" then I'm afraid it will fail either on Linux or on Windows but cannot pass on both at the same time. Then we can discuss whether we make the test Windows-specific like I did in #928, or whether we change west behavior to make the output the same on both. I will work on either option.

I'm quite uncertain what you mean here. My understanding of the situation is that I happened to be developing on Windows, and that ended up finding an inconsistency in the way West has been handling filepaths. I still don't really consider it in the scope of this PR to test or fix this inconsistency - my goal here is to fix the bug described by the original issue.

[marc-hb]

I just tried to get another look at this, really sorry for the wait.

I see the more recent commits that you added for the issue that you found and described in this comment: #920 (comment)

But the sequence of test additions and fixes in the git log didn't completely make sense to me. So, I tried to run the tests on each commit separately and I found that only the last commit https://github.com/zephyrproject-rtos/west/commits/e4d0866ee45b0076218417084be48b33b0b304ed is passing all the tests. Most other commits fail like this:

.
.
.
.
FAILED tests/test_project.py::test_update_some_with_imports - AssertionError: imap_or_mfpath has unexpected type <class 'NoneType'>
FAILED tests/test_project.py::test_update_recovery - AssertionError: imap_or_mfpath has unexpected type <class 'NoneType'>
FAILED tests/test_project.py::test_import_project_release_dir - AssertionError: imap_or_mfpath has unexpected type <class 'NoneType'>
FAILED tests/test_project.py::test_import_project_release_fork - AssertionError: imap_or_mfpath has unexpected type <class 'NoneType'>
FAILED tests/test_project.py::test_import_project_release - AssertionError: imap_or_mfpath has unexpected type <class 'NoneType'>
FAILED tests/test_project.py::test_import_project_rolling - AssertionError: imap_or_mfpath has unexpected type <class 'NoneType'>
=========== 23 failed, 272 passed, 1 xfailed in 27.29s ==============================

Every git commit must pass all its tests.

"Test-driven" development and writing tests first is great but for git bisectability and other history reasons, the order should be flipped before submission and features should come first and tests second in the end. Or all in the same commit if small enough, that's less important as long as all commits pass.

This being said, flipping the order is probably not enough in this case because a lot of the failing tests are untouched by this PR? No commit should knowingly introduce bugs fixed later in the same PR:

https://docs.zephyrproject.org/latest/contribute/contributor_expectations.html#pr-requirements

As described in this page, commits should be as small as possible... but not smaller.

[marc-hb]

My understanding of the situation is that I happened to be developing on Windows, and that ended up finding an inconsistency in the way West has been handling filepaths. I still don't really consider it in the scope of this PR to test or fix this inconsistency - my goal here is to fix the bug described by the original issue.

I could be convinced that these are truly independent of each other, but my current impression is that the Windows/Unix inconsistencie(s) when handling this path were so far hidden by the bug that this PR is fixing.

Car analogy: as long as the engine wasn't even starting, no one really cared that Windows drove on the left and Unix on the right. Cause they weren't even driving at all in the first place. Now does that mean you can fix and start the engine and still not care about the inconsistency? I would say "no" but again maybe I'm simplifying things too much and maybe my analogy is wrong.

[nmunnich]

Very sorry about the messy commits in the previous force-push. I took care to be a bit more rigorous this time, since I have more time available again. In the last one I made a silly mistake and had one crucial line which should have been in the second commit in the fourth commit.... Each commit should have all tests passing.

Regarding our inconsistency, you are in the end correct that it does seem to affect this after all. My understanding was that we have:

• the west file itself
• the output of west manifest --resolve, i.e. what west sees
• the ability of west to find and run the files found by the manifest

The previous discussion appeared when we were discussing west turning 1 into 2, where both Linux and Windows seem to behave perfectly fine regardless of what type of paths we choose, and hence I thought we were safe here. Going from 2 to 3, we see an issue appearing in Linux where it cannot interpret the windows style paths in the resolved manifest correctly.

I've added a windows-only test in the last commit which shows this. If the test is run on Linux it fails, on windows it passes.

Personally I was considering the parsing of 1 into 2 not normalising paths into posix-style to be a bug until #928. I would still consider that to be the easiest and most "proper" fix, especially since other paths in west are normalised in that manner. At the same time I feel I don't have enough stakes in the project or a weighty enough voice to get an agreement on that, hence I was really hoping we could just ignore it :(

[nmunnich]

Tried running uv run poe all inside of a Python3 vscode devcontainer to hopefully fix whatever metadata/encoding issue ruff was complaining about. The diff of the last forcepush looks very awkward, the contents are completely identical....