importing pip at runtime breaks Setuptools' distutils hack

[jaraco]

In jaraco/jaraco.develop#31 (comment), I learned that the pip outer detection is causing problems with the distutils hack.

build/src/build/env.py

Lines 156 to 176 in cc2d642

	@functools.cached_property
	def _has_valid_outer_pip(self) -> bool | None:
	"""
	This checks for a valid global pip. Returns None if pip is missing, False
	if pip is too old, and True if it can be used.
	"""
	# Version to have added the `--python` option.
	if not _has_dependency('pip', '22.3'): # pragma: no cover
	return False
	# `pip install --python` is nonfunctional on Gentoo debundled pip.
	# Detect that by checking if pip._vendor` module exists. However,
	# searching for pip could yield warnings from _distutils_hack,
	# so silence them.
	with warnings.catch_warnings():
	warnings.simplefilter('ignore')
	if importlib.util.find_spec('pip._vendor') is None:
	return False # pragma: no cover
	return True

By importing pip._vendor, build is triggering the condition that the distutils_hack uses to infer that it's running under pip (and thus should disable itself per pypa/pip#8761). Later, when setuptools is imported, it fails its assertion that distutils should be loaded locally. See the aforementioned comment for more details.

I'm trying to figure out the right fix here.

For years, Setuptools has been able to rely on the import of pip modules to detect if pip is running or not, but now that build is importing pip, it's breaking that assumption. With 35d86b8, that's no longer the case.

I can imagine how Setuptools could add additional detection logic to avoid disabling its behavior when build is importing pip, but even if I released that change today, it'll take some time to work through the ecosystem (e.g. show up in virtualenv).

My instinct is that build is violating packaging expectations, as pip is never meant to be imported, but by searching for the spec for pip._vendor, it's implicitly importing pip. But also, I can't think of a better way to honor the Gentoo use-case (#861).

What do you think, @henryiii ?

[layday]

If Gentoo is gonna modify pip, why can't they modify its dist metadata to indicate that? For example, they could add a local version label "debundled" and build could check for its existence.

[henryiii]

Does Gentoo care about Python 3.12 and older? We could make this conditional, as it's only a problem on older Python (3.11 and earlier).

[jaraco]

@mgorny ^

[mgorny]

Yeah, we still support Python 3.11, and given that PyPy is stuck at 3.11 for the foreseeable future, we're unlikely to drop support anytime soon.

I see no immediate problem with mangling the version number, but I'm worried that the presence of local versions may break badly written tools.

[henryiii]

Where does the breakage happen? Could we unload the pip module so that it doesn't see it's been loaded, or is the patching happening on module load? Edit: looks like pip's removing the shim, but we could re-add it?

[henryiii]

How do I switch from Python 3.13 to 3.11 when installing pip with emerge? I'm using the gentoo/python docker image and it insists on 3.13 when I do emerge install python-dev/pip even though 3.10+ is installed in the image.

[mgorny]

@henryiii, e.g. PYTHON_TARGETS=python3_11 emerge -1v dev-python/pip. You may also specify multiple targets. When multiple targets are installed, you can switch the "default" version via EPYTHON=python3.11 pip ..., or python3.11 -m pip ... etc.

[henryiii]

I tried that on the gentoo/python container, and just now on the gentoo/stage3 container, and I still get a slot conflict. It seems to have 3.13 already and I can't remove it.

$ docker run -it gentoo/stage3
# emerge --sync
# emerge dev-lang/python:3.11
# PYTHON_TARGETS=python3_11 emerge -1v dev-python/pip

dev-python/requests:0
dev-python/urllib3:0
dev-python/pysocks:0
dev-python/idna:0
dev-python/charset-normalizer:0
dev-python/certifi:0

  (dev-python/certifi-3024.7.22:0/0::gentoo, ebuild scheduled for merge) USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_11 (-pypy3_11) -python3_12 -python3_13 -python3_14" pulled in by
    >=dev-python/certifi-2017.4.17[python_targets_pypy3_11(-)?,python_targets_python3_11(-)?,python_targets_python3_12(-)?,python_targets_python3_13(-)?,python_targets_python3_14(-)?] required by (dev-python/requests-2.32.5:0/0::gentoo, ebuild scheduled for merge) USE="(test-rust) -socks5 -test" ABI_X86="(64)" PYTHON_TARGETS="python3_11 (-pypy3_11) -python3_12 -python3_13 -python3_14"


  (dev-python/certifi-3024.7.22-5:0/0::gentoo, installed) USE="-test" ABI_X86="(64)" PYTHON_TARGETS="python3_13 (-pypy3_11) -python3_11 -python3_12 -python3_14" pulled in by
    >=dev-python/certifi-2017.4.17[python_targets_pypy3_11(-)?,python_targets_python3_11(-)?,python_targets_python3_12(-)?,python_targets_python3_13(-)?,python_targets_python3_14(-)?] required by (dev-python/requests-2.32.5-1:0/0::gentoo, installed) USE="(test-rust) -socks5 -test" ABI_X86="(64)" PYTHON_TARGETS="python3_13 (-pypy3_11) -python3_11 -python3_12 -python3_14"

Edit: multiple targets seems to be working (at least it's building stuff), so trying that.

[henryiii]

Okay, takes a while, but using multiple targets works PYTHON_TARGETS="python3_11 python3_13"). I tried importlib.metadata.distribution("pip").files, but it seems Gentoo doesn't include proper distribution metadata, this returns an empty list. I rather thought it might.

The problem lies in the fact find_spec("pip.<thing>") has to import pip; what we could do is something like this:

         if (spec := importlib.util.find_spec('pip')) is not None:
             for loc in spec.submodule_search_locations:
                 vendor_loc = Path(loc) / "_vendor"
                 if vendor.is_dir():
                     break
             else:
                return False  # pragma: no cover 

This is safe, I believe, since it doesn't import pip when you look for the pip package itself. This shouldn't trigger warnings, either. It would not work if this wasn't a simple filesystem (like a zipapp), but I think that's okay?

[henryiii]

Could Gentoo write out a RECORD file? Detecting that seems a lot safer than these sorts of searches. If it did, we could quickly check to see if _vendor is supplied.

We could assume all pip's without a _vendor file recorded can't be used. That would cause some pips (like the homebrew one) to no longer work, but that might be better? And we could push homebrew, etc. to install properly. (Homebrew 3.13 is fine, actually, only 3.14 is missing RECORD)

[mgorny]

This is safe, I believe, since it doesn't import pip when you look for the pip package itself. This shouldn't trigger warnings, either. It would not work if this wasn't a simple filesystem (like a zipapp), but I think that's okay?

Yeah, that's fine for us, we're only concerned about filesystem paths.

Could Gentoo write out a RECORD file? Detecting that seems a lot safer than these sorts of searches. If it did, we could quickly check to see if _vendor is supplied.

I'm afraid that's not going to be simple, given that people have explicitly requested removing it to save space (given it's entirely redundant to Gentoo package metadata). IIRC the other reason was blocking pip uninstall from accidentally removing Gentoo packages but nowadays that's handled via EXTERNALLY-MANAGED.

[eli-schwartz]

We could assume all pip's without a _vendor file recorded can't be used. That would cause some pips (like the homebrew one) to no longer work, but that might be better? And we could push homebrew, etc. to install properly. (Homebrew 3.13 is fine, actually, only 3.14 is missing RECORD)

I would say, including RECORD is a complete nonstarter. The python ecosystem has repeatedly (going back to PEP 376 itself) indicated that deleting RECORD is "the canonical way" for system packages managed by a distro package manager to mark themselves as totally outside of pip (and others') authority/ability to modify. Despite attempts to use INSTALLER for the purpose, pip ended up IIRC deciding that they suspected nobody would write one, and declining to implement INSTALLER support.

This makes sense since RECORD is documented as existing for the purpose of installer tools being able to track installed files to uninstall them. And also explicitly calls out the linux distro use case:

If the RECORD file is missing, tools that rely on .dist-info must not attempt to uninstall or upgrade the package. (This restriction does not apply to tools that rely on other sources of information, such as system package managers in Linux distros.)

I assert it is out of spec for one to try using importlib.metadata.distribution("pip").files for non package manager purposes such as "does a module exist to be imported".

I'm afraid that's not going to be simple, given that people have explicitly requested removing it to save space (given it's entirely redundant to Gentoo package metadata). IIRC the other reason was blocking pip uninstall from accidentally removing Gentoo packages but nowadays that's handled via EXTERNALLY-MANAGED.

The former is a "nice to have", the latter is mandatory as EXTERNALLY-MANAGED is a bad joke that doesn't achieve its purpose. I and many others have EXTERNALLY-MANAGED in INSTALL_MASK (or various PM-specific equivalents) to make pip install --user work as designed. See also the standard internet suggestion to use --break-system-packages. Never design a spec with the intent that not everyone shall consider it a good spec thus the spec needs an opt-out mechanism.

[layday]

The externally-managed rant aside, do you have a suggestion on how this should be approached in build?