Cannot install a direct reference dependency as an "editable" dependency

[AustinTSchaffer]

Description

It seems that you can't install a package in editable mode, while simultaneously installing a package that depends on that package via a direct reference.

I came across this issue when splitting a project into 2 packages that both live in the same repository. Package A now depends on Package B via a direct reference. Since both are in the same repository, I tried installing both in editable mode, but pip came back with:

# pip install --use-feature=in-tree-build -e package_a/ -e package_b/
Obtaining file:///home/austin/workspace/pypa/test_project/package_a
Obtaining file:///home/austin/workspace/pypa/test_project/package_b
Processing ./package_b
ERROR: Cannot install package-a==1.0.0 and package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b)
    package-a 1.0.0 depends on package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

It seems that this action should be allowed taking the wording of the error message at face value.

I did some debugging and found that Resolution._add_to_criteria from src/pip/_vendor/resolvelib/resolvers.py raises a RequirementsConflicted exception. The resolver cannot resolve the 2 requirements below. The resolver fails to generate any candidates apparently because both requirements are instances of ExplicitRequirement.

RequirementInformation(requirement=ExplicitRequirement(EditableCandidate('file:///home/austin/workspace/pypa/test_project/project_b')), parent=None)
RequirementInformation(requirement=ExplicitRequirement(LinkCandidate('file:///home/austin/workspace/pypa/test_project/project_b')), parent=EditableCandidate('file:///home/austin/workspace/pypa/test_project/project_a'))

We have 2 workarounds for this:

1. Remove the direct reference from package_a, and just have package_a depend on any package named package_b. This is an issue because Package A won't know where to get Package B unless you install them both at the same time with pip install -e ./package_a/ -e ./package_b/. More worrying is if someone registers a package on PyPI that matches the name of our "Package B".
2. Install "Package A" first, then install "Package B" later. This is the solution we went with, since Pip now takes previously installed packages into consideration when solving an environment, and we keep the versions of dependencies of both packages relatively in sync. This does feel like it could cause problems down the road.

Expected behavior

I'm not sure what should be expected in this situation? It seems like one of these should be true:

1. Pip should allow pip install -e A -e B when A depends on B via a direct reference
2. Pip's error message for Point 1 should indicate why you can't do that
3. Setuptools should allow you to install a direct-reference package in editable mode

Point 3 doesn't feel right, but just wanted to throw that out there.

pip version

21.3.dev0 (commit: a53f888)

Python version

3.9.5

OS

Ubuntu 21.04 hirsute

How to Reproduce

Create a directory named "package_b". Create a setup.py in that directory with the contents:

from setuptools import setup
setup(name="package_b", version="1.0.0")

Create a directory named "package_a" in the same directory that contains the "package_b" directory. Create a setup.py in that directory with the contents:

from setuptools import setup
import os
package_b_dir = os.path.dirname(__file__).replace("package_a", "package_b")
setup(
    name="package_a",
    version="1.0.0",
    install_requires=[
        "package_b @ file://%s" % package_b_dir,
    ],
)

You should now have:

.
├── package_a
│   └── setup.py
└── package_b
    └── setup.py

Now run pip install -e ./package_a/ -e ./package_b/.

Output

The output is above in the description section, but pasting it here as well:


# pip install --use-feature=in-tree-build -e package_a/ -e package_b/
Obtaining file:///home/austin/workspace/pypa/test_project/package_a
Obtaining file:///home/austin/workspace/pypa/test_project/package_b
Processing ./package_b
ERROR: Cannot install package-a==1.0.0 and package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b) because these package versions have conflicting dependencies.

The conflict is caused by:
    The user requested package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b)
    package-a 1.0.0 depends on package-b 1.0.0 (from /home/austin/workspace/pypa/test_project/package_b)

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[surgura]

I can reproduce this exact same error with pip 22.0.3. Workaround 2 works but is not as nice when publishing a package.

[1-900-MIXALOT]

"Setuptools should allow you to install a direct-reference package in editable mode"

I ran into this limitation as well. Would be great to see a solution.

[alextremblay]

Is there anything that can be done to fix this?

I'd like to try and contribute a solution, but I'm a bit lost on where to start.

[pfmoore]

See my comment on #12533. I think this is the same issue, and my comments there apply here too. Thanks for the explanation of how you ended up in this situation, it's easier to see the issue with a real-world example. But I think the comment I made there remains true - pip can't ensure that the dependencies (the two repositories for the two packages involved) will stay in sync, so it's unable to guarantee that any resolution it comes up with is safe.

If you have to use this layout, then I think your two workarounds are your best option.

[alextremblay]

Hi @pfmoore, I read your comments in the linked issue, and I understand what you’re saying.

I have a couple points I think are worth consideration:

don't think it's something we'd want to do just to support a usage which is at best questionable

there is one use case for this feature which is more than just questionable, it’s a use case that’s becoming more and more prevalent, barring roadblocks like this: monorepos. I do all of my development in a monorepo, and would dearly love to not have to jump through hoops to make it work with python

pip can't ensure that the dependencies (the two repositories for the two packages involved) will stay in sync

my view (and the reason I offered to submit a PR) is that a RequirementsConflicted exception where both requirements point to the same package at the same location is not actually a conflict and should not be treated as a conflict.

In the case where one of those two requirements also happens to be editable, there should be some method (command line flag or something) to allow pip to resolve the conflict by choosing either the editable or non-editable variant of that dependency

as I said in a previous comment, I’m willing to start work on a PR, I just need some guidance on portions of the code to focus my efforts

[pfmoore]

it’s a use case that’s becoming more and more prevalent, barring roadblocks like this: monorepos

I can't really comment on the monorepo use case, as I don't use them myself, but as far as I'm aware there are a number of difficulties with them, which I think need to be discussed at a higher level, to establish what the "monorepo workflow" is and what needs to change in the packaging ecosystem to properly support them - assuming we want to¹. The need to support relative pathnames as a legitimate dependency (or package) specifier is the most obvious example, and this is part of that question.

I’m willing to start work on a PR, I just need some guidance on portions of the code to focus my efforts

The problem here is that you'd need to start by looking at how pip decides whether two packages are "the same". That's a very basic decision that impacts a lot of the logic, so you could end up going down something of a rabbit hole. The basic answer is that two "candidates" (to use the term the resolver works with) are the same if they have the same package name and version. However, layered on top of that are a bunch of heuristics around source trees identified by URLs or local paths. And editable installs are another layer on top of all that, which are not really considered in the resolution algorithm at all². By all means take a look, but please don't be surprised if the problem turns out to be a lot more complex than you'd hoped.

Also, when you say "both requirements point to the same package at the same location" you need to consider how you define "the same location" - are C:\Work\myproject and T:\myproject the same location? Does your answer change if T: is mapped to C:\Work? What about c:/work/myproject or file:///c:/work/myproject? I could continue, but my point is that someone will no doubt raise an issue if we don't take account of these things - and who's to say their use case is less important than yours?

Sorry if this sounds negative. It's sort of what I was trying to get at when I said we're "doing the best we can" 🙂 I appreciate the offer of help, it's just that the learning curve is steep, and this isn't an easy place to start...

Footnotes

1. I say that on the basis that I don't know if packaging tools in other language ecosystems support monorepos, or if they are a uniquely Python workflow. ↩

2. I was one of the developers of the new resolution algorithm, and from what I recall, there is nothing in that code which takes account of the "editable" flag. ↩

[AustinTSchaffer]

@pfmoore thanks so much for all of the details and background on this issue. I've also been working with monorepos more across various languages, and I think this issue was a result of me making a foray into that line of thinking.

I think in the future I'll just avoid doing the part that got me into this mess:

I came across this issue when splitting a project into 2 packages that both live in the same repository.

I honestly don't really remember what I was trying to accomplish by doing this. Maybe dependency isolation?

I think in the future I'll probably just use a single package definition, instead of trying to divide one large package into components. The "optional-dependencies" feature would allow downstream applications to reduce the total number of packages which are installed, based on need. Also there's no reason why the applications themselves couldn't live in the same repository under the same package structure, with different entrypoints defined via project.scripts.

The workaround that I actually ended up going with was to not try installing the package(s) in editable mode. If I made changes to one of the underlying packages while working on one of the downstream applications, I just reinstalled the dependencies locally after making changes to them. I also got a new job shortly afterward, which helped as well.