PEP 440 handling of prereleases for Specifier.contains, SpecifierSet.contains, and SepcifierSet.filter

[notatallshaw]

Fixes #854
Fixes #895
Fixes #856
Fixes #917

Builds on top of and, if accepted, supersedes #872

The goal of this is to have packaging specifier methods consistently follow the recommendation of PEP 440 and the specification:

• accept already installed pre-releases for all version specifiers
• accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier
• exclude all other pre-releases from consideration

Specifically this PR fixes Specifier.contains, SpecifierSet.contains, and SepcifierSet.filter, but does not change the behavior of Sepcifier.filter which already correctly complies with the above points.

This PR moves all the logic of implementing this into the filter methods and then makes the contains methods fairly simple wrappers around the filter methods.

This PR adds many tests, but could probably do with thousands more.

[notatallshaw]

Okay, I've reviewed this, I'm open to more simplification but I don't currently see anywhere obvious.

@henryiii I think you asked me at the packaging summit if there had been any discussion on this, unfortunately the discussion on interpreting the pre-release specification comes from many scattered sources.

My ultimate motivation here is that I would like a standards compliant fast resolver writen in Python. And there are certain logical operations you need to be able to do with specifiers to write fast resolvers, such as taking the intersection of two specifiers, or having a canonical normal form of a specifier.

Because of pre-releases these operations aren't obvious even when following the spec, but because packaging doesn't consistently follow the spec it's currently impossible to implement these with packaging right now. There are a couple of discussions at:

• Add a method to simplify a SpecifierSet #776
• And at: https://discuss.python.org/t/proposal-intersect-and-disjoint-operations-for-python-version-specifiers/71888

These conversations always get stuck because there is a confusion about how SHOULD pre-releases behave, so I have carefully spent time to understand what pre-releases SHOULD do. Feel free to ask any questions about this.

But this does also fix a bunch of pip issues (pypa/pip#7579, pypa/pip#9121, pypa/pip#12469).

[notatallshaw]

BTW, here is a specific invariant that this will fix that is required for certain resolution algorithms:

• Given specifier A results in no available versions, then specifier A and specifier B also results in no available versions

This does not currently hold for packaging SpecifierSets:

>>> list(SpecifierSet('<=2').filter(['1.0a1']))
[]
>>> list(SpecifierSet('<=2,<=4b1').filter(['1.0a1']))
['1.0a1']

But it SHOULD according to this line in the spec:

accept remotely available pre-releases for version specifiers where there is no final or post release that satisfies the version specifier

[henryiii]

I'll merge next week unless there are objections!

[notatallshaw]

I'm going to ping @pradyunsg who asked by Brett to review my previous PR #794 that affected pre-releases, in case they have any input.

Also in case they have any thoughts, I'm going to ping @pfmoore, as I was very much inspired to raise this PR by our discussion in this thread: https://discuss.python.org/t/proposal-intersect-and-disjoint-operations-for-python-version-specifiers/71888

I will also note, for tools using packaging to determine if a pre-release should be selected this will make that more likely, such as in pip, and such tools may want to give control to the user to prevent pre-releases (for pip I have an open issue pypa/pip#13221). When this lands in pip I will do a careful review of how pip is using Specifier and SpecifierSet as there are likely code workarounds from legacy behavior.

[pfmoore]

I'm going to ping @pfmoore, as I was very much inspired to raise this PR by our discussion in this thread

I'll note that although I was arguing in that thread that the existing spec isn't ambiguous, and tools that don't implement the spec as written are wrong (as opposed to having an "interpretation" of ambiguous wording), I'm not actually a big fan of the current spec¹. The whole pre-release handling behaviour is far too much a case of "do what I mean" for my liking. I think we would be much better off with a simpler, clearer spec, even if that meant people had to be more explicit about whether they wanted pre-releases. Of course, that would require a new PEP, rather than just a PR to the packaging project...

All of which is to say that if this change causes incompatible behaviour changes in pip, I'd be very sympathetic with users who objected. "The spec made us do it" is a bit of a cop-out IMO. But let's wait and see what the review comes up with - I don't want to start a panic for no reason.

Footnotes

1. I was involved in the original PEP that defined the specifier matching behaviour, so I'm as much to blame as anyone for the way the spec is worded - I'm not trying to point the finger at anyone here! ↩

[notatallshaw]

I think we would be much better off with a simpler, clearer spec, even if that meant people had to be more explicit about whether they wanted pre-releases. Of course, that would require a new PEP, rather than just a PR to the packaging project...

I would happily support someone else to work on that 😉. I'm of the opinion that having a spec become more mathematically simple would be a big breaking change (namely < V would have to change so it could select a pre-release version of V) for all installers / resolvers , and therefore would need buy-in from all popular package installers.

At the moment, packaging (and by extension pip) almost follows the spec, Poetry does follow the spec, uv has made design choices here that are explictly not following the spec. So I would rather just fix packaging to follow the spec, and deal with future spec changes independently.

All of which is to say that if this change causes incompatible behaviour changes in pip, I'd be very sympathetic with users who objected. "The spec made us do it" is a bit of a cop-out IMO. But let's wait and see what the review comes up with - I don't want to start a panic for no reason.

I agree with that this should not be motivated by implementing the spec for specs sake.

But this causes real problems in pip (pypa/pip#7579, pypa/pip#9121) and packaging (#854), and as mentioned it breaks important invariants for implementing more advanced resolution algorithms (#897 (comment)).

Further, packaging implements the spec correctly in some places (Specifier.filter and empty SpecifierSet.filter) but not others ( Specifier.contains, SpecifierSet.contains, and non-empty SepcifierSet.filter) e.g. #856 and #917. So depending on how the downstream tool (like pip) uses packaging, they may end up following the spec, not following the spec, or following it some times but not others.

I think these all build a strong case for changing the behavior, I also think the real world impact will be fairly small, there was very little fall out from #794.