Add support for abi3.abi3t tag from PEP 803

[ngoldbaum]

This updates the tags logic to handle abi3t from PEP 803. Also adds tests to validate everything.

Opening as a draft for visibility and to aid end-to-end testing of PEP 803. See https://github.com/Quansight-Labs/stable-abi-testing.

[pfmoore]

The tags returned here should all be simple tags, not compound ones (i.e., no dots in the components).

[pfmoore]

Same here - no dots in returned tags.

[ngoldbaum]

@pfmoore, thank you! I also pushed an accompanying fix to my setuptools PR that allows setuptools to correctly support compressed ABI tags without incorrectly returning compressed tags from cpython_tags here. See the newest commit in pypa/setuptools#5193.

[pfmoore]

Just to be 100% clear here, when you returned abi3.abi3t, you did mean that the environment will work with either an abi3 or an abi3t wheel, didn't you?

You seem now to be saying that free threaded builds only support abi3t.

I think one of us is confused here 🙁

[ngoldbaum]

You seem now to be saying that free threaded builds only support abi3t.

Take a look at the test I added that checks the output of tags.cpython_tags((3, 16), ["cp316"], ["platform"] and note how there are abi3t tags for both 3.15 and 3.16.

https://github.com/pypa/packaging/pull/1099/changes#diff-c2a8158c2ea75e325dc5d418fe408374d215b771f40be4a907dca380ad4a6701R1060-R1080

It turns out setuptools doesn't currently support compressed ABI tags unless I make packaging produce them, which is why I thought that was necessary initially. I also had to slightly patch how setuptools checks whether tags are compatible by calling tags.parse_tags. See the change I made to the assert here: https://github.com/pypa/setuptools/pull/5193/changes#diff-5f0d2f159393c18a35b50b6a97212ace4cb44ef75df0c85dd28edb23dce9a782L356

[pfmoore]

OK, some thoughts. But I'm getting way out of my depth here, and I'm concerned that we're not going off the PEP here but rather trying to define behaviours that aren't clearly stated in the PEP. It's quite likely we need to get the PEP clarified before making decisions here.

For example, why is cp315-abi3t so low in the priority order in the cp316 example? Surely a cp315-abi3t wheel should be preferred over a cp310-abi3 one? Or is the ordering only implemented in sys_tags() and the low-level functions don't need to yield their values in preference order? This is where I don't understand the packaging implementation well enough to know what's acceptable - but I'm 99.9% sure that pip calls cpython_tags directly and relies on the values being returned in "most preferred first" order...

I can't comment on the setuptools point. But I do know that pip won't work if packaging returns a compressed tag.

I feel that this might need to be brought back to Discourse. At the very least, I think we need more eyes on the problem.

[ngoldbaum]

For example, why is cp315-abi3t so low in the priority order in the cp316 example?

Because it simply didn't occur to me that sys_tags is in priority order. But also it says it right there in the docstring:

packaging/src/packaging/tags.py

Lines 654 to 660 in cd1520c

	def sys_tags(*, warn: bool = False) -> Iterator[Tag]:
	"""
	Returns the sequence of tag triples for the running interpreter.
	The order of the sequence corresponds to priority order for the
	interpreter, from most to least important.
	"""

I adjusted the logic in my latest commit.

I feel that this might need to be brought back to Discourse. At the very least, I think we need more eyes on the problem.

I agree, let's ask Petr to clarify the tag priority order for the GIL-enabled build, since it's ambiguous in PEP 803 right now.

[ngoldbaum]

And after trying to draft a message about this, I think I finally understand what needs to happen here.

Installers should only install wheels on the free-threaded build with an abi3t tag, that means packaging should only produce abi3t tags on the free-threaded build.

I was confused in my original implementation, because I thought that packaging was responsible for generating compressed tag sets, but that's wrong. Build tools do that.

I was also confused by the fact that setuptools didn't handle compressed tag sets in abi tags. I'll raise this issue with the setuptools maintainers separately from PEP 803 support.

I agree that this is a little "sparely" described in the PEP. I'll post a followup to discourse about this.

[encukou]

Suggested change

	if use_abi3t and minor_version > 14:
	if use_abi3t:

[edit: practically this could be minor_version >= 13, but it's not worth the special case.]

[pfmoore]

Given the clarification in the Discourse thread, I'd strongly recommend adding some sort of test(s) here to validate that if you change the Python implementation from 316 to 316t, the list of tags remains unchanged except for abi3 changing to abi3t and cp316 changing to cp316t. And probably vice versa, for good measure 🙂

[pfmoore]

The Discourse discussion suggests that cp314-abi3t, cp313-abi3t, cp312-abi3t, ... should also be returned here...

[ngoldbaum]

I think the last push responds to all comments. Thanks for helping me to understand this better!

[henryiii]

This looks reasonable to me.