support 'west init --topdir'

[thorsten-klein]

Fixes #774 (Successor of #775)

This PR introduces a new --topdir option for west init, allowing users to explicitly specify the west workspace directory (topdir) during initialization.

This is particularly useful when using --local, as the current behavior always sets the parent of the given directory as the topdir. In addition, west is currently unable to determine both manifest.path and manifest.file from a single directory argument as the path can be interpreted in multiple ways (e.g. it can be split at various places).

With the new --topdir argument, users can now directly control where the west workspace is created.

Parameterized tests for west init --topdir are added as well.

Moreover, the help texts and descriptions are (hopefully) simplified.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.52%. Comparing base (a4ce10a) to head (8e07584).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #854      +/-   ##
==========================================
+ Coverage   84.40%   84.52%   +0.12%     
==========================================
  Files          11       11              
  Lines        3385     3405      +20     
==========================================
+ Hits         2857     2878      +21     
+ Misses        528      527       -1     

Files with missing lines	Coverage Δ
src/west/app/project.py	80.74% <100.00%> (+0.48%)	⬆️

[mbolivar]

Seems reasonable to me

[pdgendt]

Looks good!

[pdgendt]

@marc-hb I'm not sure if @fouge is very active? I think we can move forward with this?

[marc-hb]

I really want to review this one because I spent a lot of time on #774 and #775 but @thorsten-klein is just submitting way too much code, I can't keep up.

west did not use to be like this, help! :-D

Can you give me a few more days?

[thorsten-klein]

@thorsten-klein is just submitting way too much code, I can't keep up.

Sorry 🤞

Can you give me a few more days?

Yes, sure 👍🏻 Thank you in advance for your time and input!

[carlescufi]

@thorsten-klein is just submitting way too much code, I can't keep up.

Seconded, and I'll take this opportunity to thank you for all your contributions @thorsten-klein.

[carlescufi]

Even though I'd prefer —topdir was only compatible with -l, this is not a blocking issue for me. Thanks!

[pdgendt]

Let's wait for feedback from @marc-hb, as he requested some more time.

[marc-hb]

This is particularly useful when using --local

I downloaded this PR and tested the other, --manifest-url option and it does not seem to work for me. I ran out of time and did not try --local yet. I didn't look at the code either.

Describing directory nesting clearly and accurately is not easy - it is basically a "specification" except it has to be user-friendly too. So I will look at the code only when that "specification" is complete and unambiguous.

This help looks like the two last ones can be both used at the same time:

west init [-m URL] [--mr REVISION] [--mf FILE] [-o=GIT_CLONE_OPTION] [-t WORKSPACE_DIR] [directory]

But it's not actually possible:

FATAL ERROR: --topdir cannot be combined with positional argument [directory]

When using either separately, each one seems to have the same effect. So, there does not seem to be any new feature in the remote case, which means it's still not possible to west init a manifest two levels down or more in the remote case? Is this an oversight? The help does not seem to say. I don't see why --local would be more flexible than --manifest-url. I would hope the --manifest-url and --local options work the same immediately after the git clone operation.

[marc-hb]

This is particularly useful when using --local

I downloaded this PR and tested the other, --manifest-url option and it does not seem to work for me. I ran out of time and did not try --local yet. I didn't look at the code either.

Describing directory nesting clearly and accurately is not easy - it is basically a "specification" except it has to be user-friendly too. So I will look at the code only when that "specification" is complete and unambiguous (EDIT and of course: working for me).

This help looks like the two last ones can be both used at the same time:

west init [-m URL] [--mr REVISION] [--mf FILE] [-o=GIT_CLONE_OPTION] [-t WORKSPACE_DIR] [directory]

But it's not actually possible:

FATAL ERROR: --topdir cannot be combined with positional argument [directory]

When using either separately, each one seems to have the same effect. So, there does not seem to be any new feature in the remote case, which means it's still not possible to west init a manifest two levels down or more in the remote case? Is this an oversight? The help does not seem to say. I don't see why --local would be more flexible than --manifest-url. I would hope the --manifest-url and --local options work the same immediately after the git clone operation.

[thorsten-klein]

Thank you for the review 🙏

When using either separately, each one seems to have the same effect. So, there does not seem to be any new feature in the remote case, which means it's still not possible to west init a manifest two levels down or more in the remote case? Is this an oversight? The help does not seem to say. I don't see why --local would be more flexible than --manifest-url. I would hope the --manifest-url and --local options work the same immediately after the git clone operation.

Yes, exactly. --topdir and positional argument directory do exactly the same in remote case.
I have proposed to get rid of this positional argument at all, but it will not be accepted for now because it is a breaking change (what I fully understand).

Nevertheless, in remote case those arguments that do the same are mutually exclusive (see error message you got).
I have applied your suggested change, which totally make sense to point out in the help that they both cannot be combined ([--topdir DIR | directory]) 👍🏻

it's still not possible to west init a manifest two levels down or more in the remote case?

This has already worked before in remote case, since the positional argument [directory] already exists and it does exactly the same as --topdir (see above). The main value in this change lays in the local case (and is the pre-work to potentially remove the positional argument [directory] in future)

Thank you 👍🏻

[thorsten-klein]

I think I have resolved all review comments in the code👍🏻

[marc-hb]

it's still not possible to west init a manifest two levels down or more in the remote case?

This has already worked before in remote case, ...

Has it? Please share one example of a west init --manifest-url ... command that can produce this:

workspace
├── extra
│   └── level
│       └── manifest_clone
│           └── .git
└── .west

[thorsten-klein]

it's still not possible to west init a manifest two levels down

Yes, you are right. I will push my changes where I have added support for this and also some test.

[marc-hb]

I gave the new version a try and I think I found two confirmed bugs, see below. Reproducing and describing these bugs was very time-consuming so I ran out of time again and I didn't look at the code again :-(

I love the energy but please be much more thorough in testing edge cases and addressing review comments, otherwise the limited review bandwidth cannot keep up. It's a bit counter-intuitive but in this case "slowing down" means finishing sooner.

There seems to be a large quantity of test code being added which is great! However the "quality" of the test code matters even more than its quantity. I mean being imaginative and trying hard to "break" the production code by reaching as many edge cases as possible is much more important than being repetitive. Too much test repetition only makes running the test suite longer.

Bug 1: spurious level in remote --topdir

# west init  --topdir ws/   ws/manifest_dir/

$ tree -a -L 3 ws

ws
├── manifest_dir
│   └── zephyr      # <=== !!
│     ├── .git
│     
└── .west
    └── config

I don't think this extra "zephyr" level matches the current --help text. Either way, I don't think the extra level is the correct behavior because the extra level is inconsistent with how the --local behavior treat the positional argument.

Funny enough, this rejoins my other observation that writing "workspace/manifest created 'in' X/" is ambiguous and a bad idea. Does "in" mean "X/.git" or "X/manifest/.git"? Unclear, avoid "in".

Bug 2: re-initialization is not always caught

I suspect the check that catches the re-initialization needs to be "upgraded" for the new --topdir

west init ws

cd ws
west init --local zephyr
FATAL ERROR: already initialized in ...                       # Good!

cd ..
west init --local --topdir ws/  ws/zephyr/
=== Initializing from existing manifest repository zephyr     # What?
--- Creating ws/.west and local configuration file
=== Initialized. Now run "west update" inside ws.

Interestingly enough, .west/config does not seem to be modified - unlike what the logs pretend. So there may be more than one bug there. I mean check why the logs are lying BEFORE fixing the check.

[marc-hb]

Suggested change

	path from the git clone to the workspace).
	path to the git clone from the workspace).

[marc-hb]

Maybe that's because I'm not a native speaker (anyone?) but I still read this as "the path (that leads) to the workspace", which obviously does not sound right.

Suggested change

	point at the cloned directory (relative path to the workspace).
	point at the cloned directory (relative path from the workspace).

[thorsten-klein]

I chose naming analogue to pathlib where it also says .relative_to(other)
Ref: https://docs.python.org/3/library/pathlib.html

[marc-hb]

Suggested change

	Defaults to `{_WEST_YML}` if not provided.
	Defaults to '{_WEST_YML}' if not provided.

Help texts generally don't use backquotes. I had a look and I don't think west has been so far.

Same issue below.

[thorsten-klein]

I think there are already some backslashes in the help of ManifestCommand (in this same file)

This is similar to `git status` for untracked files

Do you nevertheless prefer single quotes?

[thorsten-klein]

I suspect the check that catches the re-initialization needs to be "upgraded" for the new --topdir

@marc-hb I’ve tried with the code to fail if the topdir is already initialized (when using --local).
However, it seems that some tests (e.g. test_update_some_with_imports) rely on the current behavior and perform a re-initialization of some already initialized workspace.
What shall I do?
It looks like this behavior might have been intentional, so I make it fail only when the new argument --topdir is used.

[thorsten-klein]

Has it? Please share one example of a west init --manifest-url ... command that can produce this

workspace
├── extra
│   └── level
│       └── manifest_clone
│           └── .git
└── .west

You requested that west is able to create such a layout. The command for this is now:
west init --topdir ws ws/extra/level

All projects will be cloned under ws/extra/level.
Is this correct behavior or what would be your expectation when both args.topdir and directory are provided?
If they are mutually exclusive (as requested above), your requested feature will not be possible.
Currently. they are not mutually exclusive anymore in order to support your use case.