How to deal with namespace-packages that are split between multiple repos?

[TalAmuyal]

We have multiple repos in which we define packages under the internal namespace.
In a single-package repo, we produce a package called internal.db (where internal is a namespace package).
In the mono-repo where I try to integrate Pants, internal.db is in use and other packages under the internal namespace are defined.

I think that there is a conflict caused by the fact that internal is partially defined inside the mono-repo and partially outside.

The error I get is of the following form (I get lots of them and they come in pairs):

15:44:04.72 [WARN] The target internal_backoffice/internal/backoffice/__init__.py:../../internal-backoffice-src-sources imports `internal.db`, but Pants cannot safely infer a dependency because more than one target owns this module, so it is ambiguous which to use: ['//:root#internal-db', 'internal_analytics/internal/__init__.py:../internal-analytics-src-sources', 'internal_athena/internal/__init__.py:../internal-athena-src-sources', 'internal_data_science/internal/__init__.py:../internal-data-science-src-sources', 'internal_logs/internal/__init__.py:../internal-logs-src-sources', ...truncated..., 'internal_sqs/internal/__init__.py:../internal-sqs-src-sources'].

Please explicitly include the dependency you want in the `dependencies` field of internal_backoffice/internal/backoffice/__init__.py:../../internal-backoffice-src-sources, or ignore the ones you do not want by prefixing with `!` or `!!` so that one or no targets are left.

Alternatively, you can remove the ambiguity by deleting/changing some of the targets so that only 1 target owns this module. Refer to https://www.pantsbuild.org/v2.14/docs/troubleshooting#import-errors-and-missing-dependencies.
15:44:04.72 [WARN] Pants cannot infer owners for the following imports in the target internal_backoffice/internal/backoffice/__init__.py:../../internal-backoffice-src-sources:

  * internal.db (line: 11)

I’ve tried adding the following to the root BUILD file but it didn’t fix it:

__defaults__({
  python_requirements: {
    "module_mapping": {
      "internal-db": ["internal.db"],
    }
  }
})

And internal-db is in Pants’ req file.

I’ve made a repository with a reproduction.
Note that it is on the namespace_collision_example branch.
The README.md (linked) has the details.

Thanks in advance 🙂

[benjyw]

Thanks for posting this, and for the easy repro.

The issue here is due to this code:

pants/src/python/pants/backend/python/dependency_inference/module_mapper.py

Line 149 in 2168bb3

# If the module is not found, try the parent, if any. This is to accommodate `from`

That code exists so that we correctly infer from imports that import a symbol defined in a file. E.g., from my_project.app import App.

In this case, it is causing us to identify pacifica_db/pacifica/__init__.py as a potential provider of pacifica.cli, even though it isn't.

Will think about a solution.

[TalAmuyal]

Currently it is the only known blocker for us to adopt Pants.
Please LMK if I can do anything to help with solving it.

[TalAmuyal]

My naive solution would be to first check the parsed form of BUILD.__defaults__.python_requirements.module_mapping since it mapps internal.db by internal-db (the "internal-db": ["internal.db"] mapping) that solves it.

If it being the "default" is an issue due to shadowing, maybe we can add a BUILD.__overrides__... variation for cases that should have a higher priority.

WDYT?

[TalAmuyal]

Another option is to have a dedicated configuration in the root BUILD or pants.toml to indicate it.
Maybe something like [python].namespace_packages.add = ["ns1", "ns2"]?

[benjyw]

Yeah, a solution is forming in my mind...

[benjyw]

I think we want to propagate the fact that we checked the parent, and when we're in a situation where one provider was a "parent guess" and the other was "precise", we let the precise one win.

[benjyw]

@TalAmuyal Any interest in implementing this? We can guide you

[TalAmuyal]

Yea, gladly :)
We'll discuss it on Slack?

[benjyw]

I think maybe this issue is a better place, so there's a record of it (Slack is so ephemeral)?

[benjyw]

Almost all the changes would be in src/python/pants/backend/python/dependency_inference/module_mapper.py and its test.

Specifically:

• Create an enum with two values representing whether the mapping is of the module itself or on its parent.
• Add a field of that enum type to ModuleProvider, and populate it in all the callers
• At the top of map_module_to_address, use that information to resolve ambiguity - if there is at least one provider that is of the module itself, remove all ones that are on the module's parent, and then continue to the rest of the logic.

[benjyw]

As a refinement, possibly we want to do that stripping twice, once only for type stubs and once only for impls

[TalAmuyal]

Hi,
A small update:
I'm done doing some progress on the pants setup in our repo and I'll start working on this today.
Will update when i have some progress or blockers.

[benjyw]

This has been implemented in #17489, with thanks to @TalAmuyal !