manifest: model west-commands as per-entry records by pdgendt · Pull Request #966 · zephyrproject-rtos/west

pdgendt · 2026-06-10T18:47:49Z

Replace the WestCommandsType alias and the parallel _west_commands_manifest_dirs dict with a frozen WestCommandsEntry dataclass carrying (path, base_dir). Project.west_commands_entries is the new canonical attribute; the old west_commands becomes a deprecated read-only property. Schema accepts str / list[str] / list[{file, base-dir}] so resolved manifests roundtrip through as_yaml() without losing the base_dir needed to resolve file: paths inside specs imported from a project subdirectory.

Fixes #959
Supersedes #965

codecov · 2026-06-11T06:24:43Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.51%. Comparing base (186168a) to head (8660875).
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #966      +/-   ##
==========================================
+ Coverage   86.40%   86.51%   +0.10%     
==========================================
  Files          11       11              
  Lines        3532     3559      +27     
==========================================
+ Hits         3052     3079      +27     
  Misses        480      480

Files with missing lines	Coverage Δ
src/west/commands.py	`95.63% <100.00%> (ø)`
src/west/manifest.py	`95.28% <100.00%> (+0.12%)`	⬆️

marc-hb · 2026-06-16T15:50:35Z

I think the tests in #962 should got first because they record/"document" the existing behavior, as it was just changed. Then, this PR can adjust the tests if needed which will again document any behavior change (or none if none)

Bump the schema version and relax the west-commands type from str to any. The parsing of the west-commands is offloaded to the manifest parser to allow for maps with a file/base_dit keyset. Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>

Replace the WestCommandsType alias and the parallel _west_commands_manifest_dirs dict with a frozen WestCommandsEntry dataclass carrying (path, base_dir). Project.west_commands_entries is the new canonical attribute; the old west_commands becomes a deprecated read-only property. Schema accepts str / list[str] / list[{file, base-dir}] so resolved manifests roundtrip through as_yaml() without losing the base_dir needed to resolve `file:` paths inside specs imported from a project subdirectory. Signed-off-by: Pieter De Gendt <pieter.degendt@basalte.be>

marc-hb

I'm afraid "entry" sounds pretty vague.

We have a massive and self-inflicted name collision between:

The west-commands: field in the west.yml manifest
The "west-commands.yml" file(s) it points at
The OTHER west-commands: header line in those files!
The "west commands" descriptors in those files. "Descriptor" is an exception: theonly specific name around here? I wish we had more of these.
The "west commands" .py files that actually implement the commands
The mypy WestCommandsType
The WestCommand python class
A new "entry" class added by this PR
project.west_commands
others?

EVERYTHING is a "West command"! Unless it's an "extension", which does not help at all.

This mass collision starts in the user interface and documentation at https://docs.zephyrproject.org/latest/develop/west/extensions.html and unsurprisingly gets worse in the code. That relevant code has already given me many headaches and this one PR will obviously increase the count. I would like those collisions to ideally stop or at least start be mitigated.

Naming is hard, but this PR is definitely the right time to formalize some better terminology. I'm not implying a mass search/replace, at least not yet. But at least finalizing some "correct" names to accelerate communication, bug tracking and reviews.

To be fair, some of the above are unique enough not to need a specific name. For instance, there is only one Python file (per command), so a "west command python file" is specific enough. Not concise but good enough. On the other hand, the stuff in YAML files definitely needs more specific names. Even more so considering it's the user interface, even more so considering we have multiple YAML files.

Let me think about some terminology, just a heads-up.

marc-hb

I'm confused for now, sorry. I don't understand why this requires a schema change. Maybe I need more time to better understand the problem statement.

I think I understand the value of having some base_dir in the internal implementation but I don't understand yet why this has to be exposed in the (changed) user interface. Why can't base_dir simply be added as a prefix in the output of west manifest --resolve?

Adding that base_dir does not seem to add any value to the user interface, it looks like a pure hack to make the output of west manifest --resolve work again.

Also, do we really need to add yet another level of lists? We already have two. That bit really looks like a new feature, not a bug fix.

Let me think about some terminology, just a heads-up.

west-commands.yml is a DESCRIPTOR LIST (e.g: cmd_descs_list or longer). LIST sounds vague but I think we have no other, top-level list in this context and we can always extend it to "(command) descriptor list" in any context where it could be ambiguous. Hopefully, making a difference between the list, its file and the pointer to it from the manifest is obvious enough from the context when needed and they don't need different names.
Each DESCRIPTOR has a single pointer to one commands PYTHON FILE.
Each PYTHON FILE has a list of command CLASSES. No collision there.

marc-hb · 2026-06-19T15:37:02Z


+    ret: list[WestCommandsEntry] = []
+    for item in raw:
+        if isinstance(item, WestCommandsEntry):


In general, prefer the newer case and pattern matching. But don't go and change this before we close the high-level discussions.

marc-hb · 2026-06-19T16:06:15Z

+
+@dataclass(frozen=True)
+class WestCommandsEntry:
+    '''A single west-commands spec declared in a project.


The term "project" is also problematic :-(

For west, a project (not my choice...) is a git repo:
https://docs.zephyrproject.org/latest/develop/west/manifest.html#projects

But in most IDEs and even Zephyr sometimes, a "project" can be the workspace:

https://code.visualstudio.com/docs/editing/userinterface#_basic-layout

https://docs.zephyrproject.org/latest/develop/getting_started/index.html#get-zephyr-and-install-python-dependencies

In the following instructions the name "zephyrproject" is used for the workspace,

I think in the case the fix is easy: use either "workspace topdir" or "git top dir".

I think for West, it's clear what a project is? Maybe should just name it like that.

Suggested change

'''A single west-commands spec declared in a project.

'''A single west-commands spec declared in a West project.

marc-hb · 2026-06-19T16:08:19Z

+    wc1: list[WestCommandsEntry], wc2: list[WestCommandsEntry]
+) -> list[WestCommandsEntry]:
+    # Merge two west_commands lists, dropping any entry from wc2
+    # whose path is already in wc1 (first wins).


Such an override should print at least a DEBUG.

Sure, the existing code also did not print though.

marc-hb · 2026-06-19T16:53:41Z

+        ``base_dir`` needed to resolve Python files inside the spec when
+        the entry was promoted from a subdirectory submanifest.
+        '''
+        warnings.warn(


Why not just fail? This wasn't part of any documented API AFAICT

I was under the impression that all fields that do not start with an underscore are considered public? Anyway we don't need to break users of this.

pdgendt

I'm confused for now, sorry. I don't understand why this requires a schema change. Maybe I need more time to better understand the problem statement.

I think I understand the value of having some base_dir in the internal implementation but I don't understand yet why this has to be exposed in the (changed) user interface. Why can't base_dir simply be added as a prefix in the output of west manifest --resolve?

The problem is that west manifest --resolve outputs a single yaml file, aggregated from the imported ones. The west-commands.yml descriptor files stay in their projects untouched.
We can't simply add the prefix, that conflates two genuinely different directories:

path = location of the west-commands.yml spec file, relative to project root (e.g. mf_subdir/scripts/winsubA/west-commands.yml)
base_dir = the root that the file: entries inside that spec resolve against (e.g. mf_subdir)

The Python file resolves as base_dir / desc['file'], i.e. relative to mf_subdir, not relative to the spec file's own directory mf_subdir/scripts/winsubA/. So you can't fold base_dir into the path prefix; they point at different places.

Adding that base_dir does not seem to add any value to the user interface, it looks like a pure hack to make the output of west manifest --resolve work again.

True, base-dir is meaningless for a hand-authored manifest, but it is needed for a resolved manifest to work. I wouldn't call it a hack, but it is an internal detail surfacing in the user-facing schema.

Also, do we really need to add yet another level of lists? We already have two.

There's no level added? Currently we have the following "levels":

the manifest west-commands: field -> pointer(s) to spec file(s). Was str | list[str].
inside each west-commands.yml, the top-level west-commands: key -> list of descriptors.
inside each descriptor, commands: -> list of command classes.

This PR changes level 1 only, and it widens the element type of that existing list from str to str | {file, base-dir}.

That bit really looks like a new feature, not a bug fix.

We're not adding new features? Users can't express anything new. It is genuinely a bug fix IMO.

Let me think about some terminology, just a heads-up.

west-commands.yml is a DESCRIPTOR LIST (e.g: cmd_descs_list or longer). LIST sounds vague but I think we have no other, top-level list in this context and we can always extend it to "(command) descriptor list" in any context where it could be ambiguous. Hopefully, making a difference between the list, its file and the pointer to it from the manifest is obvious enough from the context when needed and they don't need different names.

Each DESCRIPTOR has a single pointer to one commands PYTHON FILE.

Each PYTHON FILE has a list of command CLASSES. No collision there.

I like the idea of using descriptor.

pdgendt · 2026-06-20T13:26:22Z

+
+@dataclass(frozen=True)
+class WestCommandsEntry:
+    '''A single west-commands spec declared in a project.


I think for West, it's clear what a project is? Maybe should just name it like that.

Suggested change

'''A single west-commands spec declared in a project.

'''A single west-commands spec declared in a West project.

pdgendt · 2026-06-20T13:27:15Z


+    ret: list[WestCommandsEntry] = []
+    for item in raw:
+        if isinstance(item, WestCommandsEntry):


pdgendt · 2026-06-20T13:27:41Z

+    wc1: list[WestCommandsEntry], wc2: list[WestCommandsEntry]
+) -> list[WestCommandsEntry]:
+    # Merge two west_commands lists, dropping any entry from wc2
+    # whose path is already in wc1 (first wins).


Sure, the existing code also did not print though.

pdgendt · 2026-06-20T13:29:10Z

+        ``base_dir`` needed to resolve Python files inside the spec when
+        the entry was promoted from a subdirectory submanifest.
+        '''
+        warnings.warn(


I was under the impression that all fields that do not start with an underscore are considered public? Anyway we don't need to break users of this.

pdgendt force-pushed the westcommand-type branch from 661bd0b to 02fb69f Compare June 11, 2026 06:14

pdgendt force-pushed the westcommand-type branch 2 times, most recently from ea2092e to ec491ab Compare June 11, 2026 08:30

pdgendt marked this pull request as ready for review June 11, 2026 09:04

pdgendt requested review from marc-hb and mbolivar June 11, 2026 09:04

pdgendt force-pushed the westcommand-type branch from ec491ab to c35e889 Compare June 12, 2026 08:48

pdgendt mentioned this pull request Jun 12, 2026

tests: manifest: Add parse/yaml roundtrip test for nested west-commands #965

Closed

marc-hb mentioned this pull request Jun 16, 2026

manifest --resolve normalizes west-commands: "sometimes" #961

Open

pdgendt force-pushed the westcommand-type branch from c35e889 to da49e22 Compare June 18, 2026 16:51

pdgendt force-pushed the westcommand-type branch from da49e22 to 8660875 Compare June 18, 2026 17:09

pdgendt mentioned this pull request Jun 18, 2026

check manifest --resolve in _subdir_special_chars() test #962

Merged

marc-hb reviewed Jun 19, 2026

View reviewed changes

pdgendt commented Jun 20, 2026

View reviewed changes

	'''A single west-commands spec declared in a project.
	'''A single west-commands spec declared in a West project.

Conversation

pdgendt commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

marc-hb commented Jun 16, 2026

Uh oh!

marc-hb left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

marc-hb left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pdgendt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

pdgendt commented Jun 10, 2026 •

edited

Loading

codecov Bot commented Jun 11, 2026 •

edited

Loading

marc-hb left a comment •

edited

Loading