Allow modifying remote/project urls of imported manifests

[jobasto]

Introduction

We use west to checkout all source files necessary for building a firmware and use the import feature as pictured below. This way we don't need to specify the exact version of the modules we want to use because that information is taken from the imported manifests.

project_repository (internal git server)
├─ west.yml
    ⇓ imports (filtered by name-allowlist)
    company_library_repository (internal git server)
    ├─ west.yml
        ⇓ imports
        zephyr_repository (https://github.com/zephyrproject-rtos/zephyr)
        ├─ west.yml
             ⇓ imports
             zephyr_module_1 (https://github.com/zephyrproject-rtos/zephyr_module_1)
             zephyr_module_2 (https://github.com/zephyrproject-rtos/zephyr_module_2)
             zephyr_module_3 (https://github.com/zephyrproject-rtos/zephyr_module_3)

Requirement

To ensure long term reproducibility we want to mirror all externally hosted repositories on our internal git server and use this mirror server for cloning.

Possible solutions

User/system specific git config

One way to achieve this would be to use the git url.<base>.insteadOf config option (see also #28 (comment)). However, this option is user and/or system specific but we want that the actually used servers are deducible by examining the project_repository and its imports.

Specifying git config options in manifest

Another way would be to introduce a new manifest file key that allows setting arbitrary git config options. However, west probably shouldn't set user or system wide config options so it would have to set the config option in all imported repositories.

Adding a rewrite-url key to the manifest

Probably the cleanest way would be to introduce a new key to the manifest file, e.g. rewrite-url. An example could look like

manifest:
  defaults:
    remote: zephyr
    rewrite-url:
      - from: "default from regexp"
        to: "default to regexp "

  remotes:
    - name: zephyr
      url-base: https://github.com/zephyrproject-rtos

  projects:
    - name: zephyr
      revision: xxxxxx
      path: modules/zephyr
      import:
        rewrite-url:
          - from: "from regexp 1"
            to: "to regexp 1"
          - from: "from regexp 2"
             to: "to regexp 2"
    - name: module
      revision: yyyyyyyyyyy
      path: modules/lib/module
      import:
         true # uses default rewrite-url

rewrite-url are a list of rewrite rules that will be applied to a url in order. If manifests are imported multiple levels deep the rewrite rules on the lowest level are applied first.

Open points:

• Is this full fledged implementation necessary or is there a more simple solution?
• Should we support full regexp or just match the start of the url as git does?
• Do we need project specific rewrite-urls? Would it make the implementation if we supported only rewrite-url in the defaults section?

Suggestions?

[marc-hb]

Did you take a look at

• west update: remotes are not imported from submanifests #565

[jobasto]

Thanks for pointing that out. I've overlooked this issue. But I'm not quite sure what to make of it.

The solution proposed is to use the url.<base>.insteadOf config option of git, which I had considered above. But if I see it correctly this mandates that every developer has to configure their git installation accordingly. And this changes the servers used to checkout without that fact being reflected in some west.yml.

[marc-hb]

But I'm not quite sure what to make of it.

I believe the salient point is "anything 'dynamic' in imports instantly becomes insanely complicated". Quoting @mbolivar-nordic in #565

I thought about how to treat remotes as first class objects you can inherit from sub-manifests that you import. It gets really messy and I couldn't come up with a clean set of rules for it. So the decision I made is that when you import a project, its fetch URL is completely resolved at import time.

Quoting him again in #608

in the presence of this feature, we can't count on a complete workspace, so all commands which require a manifest to work would need to be revisited and correct semantics decided upon (e.g. if I do west init + west update --skip foo + west list, and 'foo' has some imports in it, the manifest will fail to load and west list won't have all the information it needs to do its job. Should it fail? Do we want something else? Etc.)

To be fair, mirroring does not change any git version so it's not actually "dynamic". I'm afraid mirroring is unfortunate "collateral damage" of the more general complexity of changing imports on the fly.

My 2 cents, I'm merely trying (and probably failing) to follow @mbolivar-nordic's expert thoughts on this.

EDIT: https://github.com/zephyrproject-rtos/west/issues?q=label%3A%22Partial+imports%22

[marc-hb]

Is this full fledged implementation necessary or is there a more simple solution?

I'm instinctively not a fan of regular expressions and "rewrites" for this. It seems vastly overkill to just point at a different git clone. git submodules somewhat support mirrors and they don't need all that. A much simpler layer of remote indirection looks like it would be enough do to the job. It would also solve #565 and #608.

It could/should be something as simple git's insteadOf but implemented in workspace/.west/config to make it project-specific and not pollute unrelated work.

but we want that the actually used servers are deducible by examining the project_repository and its imports.

I don't understand why. Mirroring should make absolutely no functional difference; it should only affect build performance. If mirroring makes any functional difference then it's a bug.

[marc-hb]

but we want that the actually used servers are deducible by examining the project_repository and its imports.

Wait, this requirement is actually much worse than I thought: what if I want to use my own, personal/internal mirrors for maximum performance and fault tolerance? Or because my network is isolated for security reasons? I do NOT want to touch any manifest files in that case because comparing manifests is the very first step to make sure I'm building exactly the same thing as someone else - including for security reasons but not just[*]

[*] https://www.cisa.gov/sbom

[jobasto]

but we want that the actually used servers are deducible by examining the project_repository and its imports.

I don't understand why. Mirroring should make absolutely no functional difference; only performance and build reliability. If mirror makes a difference then it's a bug.

It's meant to ensure that we have everything necessary to build a project in-house so that in a couple of years from now no matter what happens we are able to reproduce the binaries. So I want to make it obvious that our internal mirror is used and I want to make it the default by putting the rewrite rule (or whatever method) on the company_library_repository level (because every application project is importing this repository).

[marc-hb]

It's meant to ensure that we have everything necessary to build a project in-house so that in a couple of years from now no matter what happens we are able to reproduce the binaries.

The biggest innovation brought by decentralized version control systems like git and mercurial is that the server does not matter. All clones are equivalent, they all have the same content (except of course for brand new work in progress not synchronized yet).

Hosting and content are now unrelated to each other.

Mirroring and backups come "for free".

In fact this "content-addressable distributed storage" idea is so good that it's being extended beyond git to pretty much everything: https://en.wikipedia.org/wiki/InterPlanetary_File_System

To "ensure you have everything necessary" specific server names do not and must not matter.

So I want to make it obvious that our internal mirror is used and...

If that mattered then there would be a flaw in your design.

[jobasto]

It's meant to ensure that we have everything necessary to build a project in-house so that in a couple of years from now no matter what happens we are able to reproduce the binaries.

The biggest innovation brought by decentralized version control systems like git and mercurial is that the server does not matter. All clones are equivalent, they all have the same content (except of course for brand new work in progress not synchronized yet).

That's true as long as one can find the repository somewhere. One could argue that this will be always the case for any valuable piece of source code. But I don't want to rely on somebody else to do the backup for me. Maybe we have different standpoints regarding this subject which explains our discussion.

So how do we proceed? Should we wait for @mbolivar-nordic to have a look at this issue?

[marc-hb]

That's true as long as one can find the repository somewhere.

Yes, and? I don't understand the point you're trying to make here sorry.

But I don't want to rely on somebody else to do the backup for me.

To perform backups I can (and I do) clone my own repos as many times as I like, I don't have to wait for someone else to do it.

[jobasto]

I try to summarize it. I hope this makes my thoughts clear.

1. I want to make sure that the application is buildable without relying on any external servers.
2. How do I do that? -> I mirror everything using our own servers where I know a backup process is in place.
3. How do I make sure that every repository I (or any colleague) need is on the mirror server? -> I make using the mirror server the default so that referencing a repo that is not mirrored will fail.
4. How do I make using the mirror the default (also for colleagues and new application projects)? -> I put a rewrite rule in the company_library_repository .

[marc-hb]

Thanks for making sure there is no misunderstanding.

My point is: 3. and 4. must not be in a manifest. The final mirroring decision must belong to a local configuration file. Having default values and sample configuration files in git is fine but the final mirroring decision must be in an (optional) local file that is not tracked in git.

Even when you and I build from different mirrors we must keep the ability to compare what we build and make sure it's the same thing all the way. This is becoming a legal requirement for obvious security reasons: https://www.cisa.gov/sbom

The very first step (not the only one) to compare our builds is to make sure we build from the same manifests and that our west status and git status are clean. Polluting manifests with mirroring details would change either git hashes or git status and break the ability to compare builds easily.

git (and mercurial, etc.) were a bit of a revolution because they totally decoupled location and version control thanks to the magic of checksumming everything. "Download from anywhere; I don't care". Git submodules kept that critical decoupling: you can mirror submodules and download them from anywhere and they still work. Git submodules come with a default location but it can be redirected in .git/config without polluting anything tracked in git.

West manifests should certainly not regress this wonderful design decision.

I make using the mirror server the default so that referencing a repo that is not mirrored will fail.

This does not prove anything. You could still have a build script downloading something without west or git knowing anything. For instance CMake's "ExternalProject" does that routinely. I'm not 100% sure but I think it has happened in Zephyr: https://github.com/zephyrproject-rtos/zephyr/issues?q=externalproject. There are many other ways to download from the Internet.

The only way to make really sure is to test your mirror while having no internet access.

[marc-hb]

This does not prove anything. You could still have a build script downloading something without west and git knowing anything.

Just remembered the Zephyr project has a west blob command (I never used it)

• Process: binary blobs in Zephyr zephyr#38570 (comment)

EDIT: here's another example of code being downloaded at build time. It's wrong but not uncommon

• Modules: TF-M: Resolve QCBOR issues with TF-M 1.7.0 zephyr#54017

[mbolivar-nordic]

Happy new year!

@marc-hb thank you for collecting all the backlogs!

I believe the salient point is "anything 'dynamic' in imports
instantly becomes insanely complicated".

Yep, the way import works internally is more complicated than people
usually expect.

The internal representations west uses to keep everything consistent
and well-defined when information (like project URLs) can come from
multiple places in the manifest or imported manifests (like 'remote'
or 'url' keys in an individual project definition, or the default
remote if neither is provided) can make some things harder to do than
people expect in particular.

To be fair, mirroring does not change any git version so it's not
actually "dynamic". I'm afraid mirroring is unfortunate "collateral
damage" of the more general complexity of changing imports on the
fly.

I thought about it a bit and we may be able to salvage some sort of
project URL override to satisfy this use case.

@jobasto:

manifest:
  defaults:
    remote: zephyr
    rewrite-url:
      - from: "default from regexp"
        to: "default to regexp "

This does not seem to address the core problem, which as I understand
it is how to selectively rewrite the URLs of imported projects, while
leaving everything else the same. I'm also not clear about the
intended semantics. I am inclined to say "no" to this part of a
proposal without a better description of what problem it solves and
how it solves it.

  projects:
    - name: zephyr
      revision: xxxxxx
      path: modules/zephyr
      import:
        rewrite-url:
          - from: "from regexp 1"
            to: "to regexp 1"
          - from: "from regexp 2"
             to: "to regexp 2"

I think that something like this is probably doable, though I share
@marc-hb 's concerns about using regular expressions.

Let me start by saying that while I don't have time to implement this
myself (my main tasks are devicetree-related at the present time), I
would be happy to provide some guidance on how I would like to see
this done.

The function that needs to be hooked is Manifest._load_projects:

west/src/west/manifest.py

Line 2073 in fc278d4

def _load_projects(self, manifest: Dict[str, Any],

This is what's responsible for taking the parsed YAML data and
creating a west.manifest.Project object, which has the final url
field that @jobasto wants to override.

We need to track any context-specific information that we may have
inherited from a "parent" manifest (so e.g.
project_repository/west.yml could provide a URL override, which then
needs to be passed downwards as we recursively import
company_library_repository/west.yml and finally
zephyr_repository/west.yml). At this last step, we will call
_load_projects on the manifest data defining the project, and we
need the "parent" rewrite data in project_repository/west.yml to be
available.

The correct place to put this information is _import_ctx:

west/src/west/manifest.py

Line 266 in fc278d4

class _import_ctx(NamedTuple):

The current import context is always available as
west.manifest.Manifest._ctx. You should therefore:

• add a field to _import_ctx with an internal representation for all
  the rewrite rules you need

• merge it with any new rewrite rules as you recursively resolve
  manifests (look at how _compose_imap_filters() is defined and used
  for examples of how west updates its import context as it resolves
  imports)

• make sure _load_projects consults this context when it creates the
  Project object, using the rewrite as necessary

@jobasto is this something you would be willing to take on?