Unexpected behavior with -r references inside -c references

[HALtheWise]

What's the problem this feature will solve?

Our company is using pip requirements files to manage dependencies in a relatively complex codebase with lots of deployment configurations. We currently use -r constraints to manage a tree of files for this purpose, but are having difficulties using -c constraints in this system.

As an example with three files:

# base.txt
package1==1.0.0

# dev.txt
-r base.txt
package2==2.0.0
package3==3.0.0

# deploy.txt
-c dev.txt
package3

The expected behavior of pip install -r deploy.txt would be to install only package3. Today, it instead installs both package1 and package3.

This happens because the -r base.txt line, even when found inside a "constraint" file, causes pip to install all packages in the referenced file, even if they were not referenced in the top-level file.

Describe the solution you'd like

When processing a constraint file specified with -c, pip should treat any -r references inside it as if they were actually just soft constraints. This clarifies the semantics of -c to treat the entire referenced tree as being a constraint, without allowing its contents to self-promote into "hard" references.

Effectively, this makes the -r reference mean "embed the contents of this other file into the current context".

Fixing this behavior would allow us to create a large number of different "deploy" configurations that only contain the precisely needed dependencies for deploying the particular service they reference, while ensuring that the versions of those dependencies match each other, and match the version in our development environment.

More generally, for large projects using pip that want to maintain a set of interrelated dependency lists, this allows those projects to decrease the amount of duplication needed in specifying pinned versions, even when the installed packages don't fit into a strict tree structure.

Alternative Solutions

1. Flattening the version specifiers
   By putting all of our version constraints in a single file, we could safely reference it with -c without accidentally installing things. This ends up being a huge file, and even with tooling to manage it, quickly gets unwieldy. It is nice to be able to use a -r tree to specify the parts of the dependency graph that are tree-like.

2. Constructing a temporary file hierarchy
   Without native pip support, we could copy our files into a separate directory, replace -r with -c as necessary, then run pip on those files. That's a mess, and detracts from the simplicity of pip install -r requirements.txt

3. External tooling
   We already use pip-compile-multi in our repository, and could fork that project to create "flattened" requirements files for each level of the tree, so we can use -c directly against that compiled flat file. This would mean that any version change would show up many places in the diff, making reviews more difficult.

[sbidoul]

When processing a constraint file specified with -c, pip should treat any -r references inside it as if they were actually just soft constraints. This clarifies the semantics of -c to treat the entire referenced tree as being a constraint, without allowing its contents to self-promote into "hard" references.

Effectively, this makes the -r reference mean "embed the contents of this other file into the current context".

I agree this would be more intuitive and versatile.

We'd need to think through the backward compatibility implication if we want to change this.

[uranusjr]

For clarification, is it currently possible to have -c in requirements or constraints files? The proposal sounds reasonable if only -r is currently allowed. Otherwise there will be many combinations we need to figure out. A part of me wants to replace -r entirely with a different deriative (say --include) since the letter does not really make sense if we turn the meaning into what’s proposed.

[HALtheWise]

Yes, you can put either -r or -c flags into files that are included via either -r or -c flags today. I'm hesitant to introduce a new flag that's almost identical in behavior to -r, but that would resolve our specific case.

[pradyunsg]

This feels like a side-track to solving the main semantic issue we have of the lack of separation between concrete and abstract dependencies. That's not a reason to not do this at all, but is the reason that I won't be volunteering to do this. :)

[sbidoul]

As it happens, I have been fiddling with requirement parsing recently, so I think the fix should be quite simple: #8821.

The question is: do we consider this as a bug fix, or as a behavior change that would require a feature flag and a deprecation period? My feeling is that it is at the same time obvious and not used by many, so we can avoid the fuss of the deprecation.

[pradyunsg]

Yea, I view constraints as a thing not-a-lot-of-people-use-today, so no deprecation should be OK.

[sbidoul]

@HALtheWise please check out the discussion in #8821, and in particular the question in #8821 (comment).