[BUG] pkg_resources.Requirement.parse changed exception class on 70.0.0

[pradyunsg]

setuptools version

70.0.0

Python version

N/A

OS

N/A

Additional environment information

N/A

Description

In setuptools 70, Requirement.parse raises a different (private) exception class coming out of the _vendor path rather than the extern path. This means that there is no way for a caller of that method to catch the raised exception without reaching into the _vendor directory.

This was changed by e999582, if I understand correctly.

Expected behavior

The exception class is not changed, or the class of the exception raised is publicly accessible.

How to Reproduce

1. Create a file with the following contents:

import warnings

warnings.simplefilter("ignore", Warning)

import setuptools
import pkg_resources  # noqa
from pkg_resources.extern.packaging.requirements import (  # noqa
    InvalidRequirement as ErrorFromExtern,
)
from pkg_resources._vendor.packaging.requirements import (  # noqa
    InvalidRequirement as ErrorFromVendor,
)

try:
    pkg_resources.Requirement.parse("a b")
except Exception as e:
    print(setuptools.__version__)
    print("  ", e.__class__)
    print("  ", isinstance(e, ErrorFromExtern))
    print("  ", isinstance(e, ErrorFromVendor))

2. Run this file in an environment with setuptools 70.0.0 and then in an environment with setuptools==69.5.1.
3. See the different class of the exception.

Output

69.5.1
   <class 'pkg_resources.extern.packaging.requirements.InvalidRequirement'>
   True
   False
70.0.0
   <class 'pkg_resources._vendor.packaging.requirements.InvalidRequirement'>
   False
   True

[Avasam]

The only change that I can think of affecting this would be https://github.com/pypa/setuptools/pull/4348/files#diff-e3d3d454fa3a072c9f46f8affa27513892fbc2d245e87f57a5927a7be851de05

[abravalheri]

This seems to be a bit of a quirk of Python's importlib/import machinery:

# v69.5.1
from pkg_resources.extern.packaging import requirements as _packaging_requirements

print(f"{_packaging_requirements=}")
print(f"{_packaging_requirements.__spec__=}")
print(f"{_packaging_requirements.__loader__=}")

# --- ouput ---
_packaging_requirements=<module 'pkg_resources.extern.packaging.requirements' from '/tmp/.venv/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/requirements.py'>
_packaging_requirements.__spec__=ModuleSpec(name='pkg_resources.extern.packaging.requirements', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f21e4dedee0>, origin='/tmp/.venv/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/requirements.py')
_packaging_requirements.__loader__=<_frozen_importlib_external.SourceFileLoader object at 0x7f21e4dedee0>

# v70.0.0
from pkg_resources.extern.packaging import requirements as _packaging_requirements

print(f"{_packaging_requirements=}")
print(f"{_packaging_requirements.__spec__=}")
print(f"{_packaging_requirements.__loader__=}")

# --- ouput ---
_packaging_requirements=<module 'pkg_resources._vendor.packaging.requirements' from '/usr/local/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/requirements.py'>
_packaging_requirements.__spec__=ModuleSpec(name='pkg_resources._vendor.packaging.requirements', loader=<_frozen_importlib_external.SourceFileLoader object at 0x7f15c2be9310>, origin='/usr/local/lib/python3.12/site-packages/pkg_resources/_vendor/packaging/requirements.py')
_packaging_requirements.__loader__=<_frozen_importlib_external.SourceFileLoader object at 0x7f15c2be9310>

(testing with docker run --rm -it python:3.12-bookworm /bin/bash)

But the diff in pkg_resources/extern/__init__.py is very minimum and it is not immediately clear why this diff causes such change in behaviour:

# git diff v69.5.1..v70.0.0 -- pkg_resources/extern/__init__.py
diff --git a/pkg_resources/extern/__init__.py b/pkg_resources/extern/__init__.py
index df96f7f26..bfb9eb8bd 100644
--- a/pkg_resources/extern/__init__.py
+++ b/pkg_resources/extern/__init__.py
@@ -70,12 +70,20 @@ class VendorImporter:
             sys.meta_path.append(self)


+# [[[cog
+# import cog
+# from tools.vendored import yield_top_level
+# names = "\n".join(f"    {x!r}," for x in yield_top_level('pkg_resources'))
+# cog.outl(f"names = (\n{names}\n)")
+# ]]]
 names = (
-    'packaging',
-    'platformdirs',
-    'jaraco',
+    'backports',
     'importlib_resources',
+    'jaraco',
     'more_itertools',
-    'backports',
+    'packaging',
+    'platformdirs',
+    'zipp',
 )
+# [[[end]]]
 VendorImporter(__name__, names).install()

Potentially a quirk in the importlib.util.spec_from_loader dynamics?

---

The interesting bit is that if we modify the test script like the following:

 import setuptools
 import pkg_resources  # noqa
-from pkg_resources.extern.packaging.requirements import (  # noqa
-    InvalidRequirement as ErrorFromExtern,
-)
-from pkg_resources._vendor.packaging.requirements import (  # noqa
-    InvalidRequirement as ErrorFromVendor,
-)
+from pkg_resources.extern.packaging import requirements as _packaging_requirements

 try:
     pkg_resources.Requirement.parse("a b")
-except Exception as e:
-    print(setuptools.__version__)
-    print("  ", e.__class__)
-    print("  ", isinstance(e, ErrorFromExtern))
-    print("  ", isinstance(e, ErrorFromVendor))
+except _packaging_requirements.InvalidRequirement as ex:
+    print(f"Error catched by {ex.__class__=}")

then it seems to "work" fine (although things are not "named" fine...):

$ python test.py
Error catched by ex.__class__=<class 'pkg_resources._vendor.packaging.requirements.InvalidRequirement'>

---

Also interesting and surprising:

>>> from pkg_resources.extern.packaging import requirements as req
>>> from pkg_resources.extern.packaging.requirements import InvalidRequirement
>>> InvalidRequirement == req.InvalidRequirement
False

This result seems to corroborate the hypothesis of quirks in the importlib.

[pradyunsg]

FWIW, I agree with @Avasam that the change that affects this is likely in pkg_resources/__init__.py.

Specifically https://github.com/pypa/setuptools/compare/v69.5.1..v70.0.0#diff-e3d3d454fa3a072c9f46f8affa27513892fbc2d245e87f57a5927a7be851de05 (I can't figure out how link to a line).

[abravalheri]

FWIW, I agree with @Avasam that the change that affects this is likely in pkg_resources/__init__.py.

Specifically https://github.com/pypa/setuptools/compare/v69.5.1..v70.0.0#diff-e3d3d454fa3a072c9f46f8affa27513892fbc2d245e87f57a5927a7be851de05 (I can't figure out how link to a line).

The problem is that it is not immediately obvious why the change:

# FROM
- from pkg_resources.extern import packaging
-
- __import__('pkg_resources.extern.packaging.version')
- __import__('pkg_resources.extern.packaging.specifiers')
- __import__('pkg_resources.extern.packaging.requirements')
- __import__('pkg_resources.extern.packaging.markers')
- __import__('pkg_resources.extern.packaging.utils')
# TO
+ from pkg_resources.extern.packaging import markers as _packaging_markers
+ from pkg_resources.extern.packaging import requirements as _packaging_requirements
+ from pkg_resources.extern.packaging import utils as _packaging_utils
+ from pkg_resources.extern.packaging import version as _packaging_version

would cause such a problem. From my point of view they should be equivalent in the context of the previous PR that introduced the change...

---

On the other hand, this is absolutely surprising:

>>> from pkg_resources.extern.packaging import requirements as req
>>> from pkg_resources.extern.packaging.requirements import InvalidRequirement
>>> InvalidRequirement == req.InvalidRequirement
False

So that is why I believe this related to the root cause of the problem.

[Avasam]

With extern gone, here's an updated example, the result is back to 69.5.1-like.

import warnings

warnings.simplefilter("ignore", Warning)

import setuptools
import pkg_resources
from packaging.requirements import InvalidRequirement as ErrorFromInstall
from setuptools._vendor.packaging.requirements import (
    InvalidRequirement as ErrorFromVendor,
)

try:
    pkg_resources.Requirement.parse("a b")
except Exception as e:
    print(setuptools.__version__)
    print("  ", e.__class__)
    print("  ", isinstance(e, ErrorFromInstall))
    print("  ", isinstance(e, ErrorFromVendor))

This result is the same with or without packaging installed:

72.1.0.post20240731
   <class 'packaging.requirements.InvalidRequirement'>
   True
   False

[abravalheri]

Thank you very much @Avasam. I think this settles the issue (since _vendor is not meant to be used directly).

@pradyunsg is this resolution enough to support your use case?