[BUG] easy_install doesn't handle sysconfig vars containing multiple space separated paths

[karlicoss]

setuptools version

setuptools==68.2.2

Python version

Python3.11 (via pyenv)

OS

macos

Additional environment information

$ python3 --version
Python 3.11.2
$ python3 -m pip --version
pip 23.2.1 from /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11/site-packages/pip (python 3.11)

Description

I'm using python3.11 via pyenv (this is the only version of python I have apart from system python3.9).
While I've never encountered this issue before, and it involves pyenv, I debugged a bit and I believe it's a setuptools rather than pyenv issue.

When trying to install any editable package, e.g. python3 -m pip install --user -e /path/to/pkg, somehow it creates an extra directory ending with a space in ~/.pyenv/versions:

$ ls -1 ~/.pyenv/versions/
3.11.2
3.11.2

(hard to see obviously because it ends with space 🙃 )

I think this causes some issues further down the line for pyenv, but it's kinda irrelevant for this bug.
I tried making ~/.pyenv/versions directory read only and running setuptools with debug information:

$ DISTUTILS_DEBUG=true python3 -m pip install --user -e /path/to/pkg
...
        Distribution.get_command_obj(): creating 'egg_info' command object
        os.makedirs('/Users/karlicos/.pyenv/versions/3.11.2 /Users/karlicos/.pyenv/versions/3.11.2/lib /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11 /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11/lib-dynload', 0o700)
        error: [Errno 13] Permission denied: '/Users/karlicos/.pyenv/versions/3.11.2 '
...

It seems that this happens here

setuptools/setuptools/command/easy_install.py

Lines 1366 to 1374 in 82f5c60

	def create_home_path(self):
	"""Create directories under ~."""
	if not self.user:
	return
	home = convert_path(os.path.expanduser("~"))
	for path in only_strs(self.config_vars.values()):
	if path.startswith(home) and not os.path.isdir(path):
	self.debug_print("os.makedirs('%s', 0o700)" % path)
	os.makedirs(path, 0o700)

The offending variable turned out to be DESTDIRS, which contained a bunch of paths separated by a space. There is also INCLDIRSTROMAKE in my sysconfig that contains multiple paths

python3 -c 'import sysconfig; print({k: v for k, v in sysconfig.get_config_vars().items() if isinstance(v, str) and v.startswith("/Users/karlicos/") and " " in v})'
{'DESTDIRS': '/Users/karlicos/.pyenv/versions/3.11.2 /Users/karlicos/.pyenv/versions/3.11.2/lib /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11 /Users/karlicos/.pyenv/versions/3.11.2/lib/python3.11/lib-dynload', 'INCLDIRSTOMAKE': '/Users/karlicos/.pyenv/versions/3.11.2/include /Users/karlicos/.pyenv/versions/3.11.2/include /Users/karlicos/.pyenv/versions/3.11.2/include/python3.11 /Users/karlicos/.pyenv/versions/3.11.2/include/python3.11'}

I'm not sure what exactly the aim of this function, but perhaps it needs to do some extra checks before trying to create dirs. I suppose it's possible that it's a misconfiguration on my side somehow, but I think the variable names suggest they may contain multiple paths.

possibly relevant issue: #3063

Expected behavior

Package is installed as editable, no extra dirs are created

How to Reproduce

Not sure how to provide consistent steps to reproduce since it requires pyenv set up in the first place, and there is no Docker on macos to try it

Output

provided above

[abravalheri]

Hi @karlicoss, thank you very much for the feedback.

We will need a reproducer to investigate this one (and also a person that has access to a macOS and is willing to help 😝¹).

Something that confuses me is that you are pointing out easy_install.
This piece of setuptools is deprecated and in theory is not used if you are trying to install a project that was created using the latest standards... Does the error still persist if you explicitly add the --use-pep517 flag to pip? (DISTUTILS_DEBUG=true python3 -m pip install --use-pep517 --user -e /path/to/pkg)

Footnotes

1. I don't have access to a macOS so I will not be able to help much in the investigation. ↩

[karlicoss]

@abravalheri thanks, if I do that with --use-pep517, it works!
It's the first time I hear about pep517 though! By 'using latest standards', do you mean using pyproject.toml or something like that instead of setup.py (it's the latter in my case).

I'll try to come up with a reproducer when I'm near a linux pc so I can try it in docker (doubt it's mac os specific).

[abravalheri]

By 'using latest standards', do you mean using pyproject.toml or something like that instead of setup.py (it's the latter in my case).

No, by 'latest standards', I just mean build isolation + PEP 517/PEP 660 backend APIs.

No one need to use pyproject.toml to configure setuptools, setup.py is perfectly fine as configuration file¹. pip>=23.1 was meant to adopt isolation by default, but the change ended up delayed to avoid incompatibilities.

Having an pyproject.toml file in the project root or adding the --use-pep517 flag are 2 different ways of forcing pip to tap into isolation.

Footnotes

1. Running python setup.py ... directly as a CLI tool is deprecated however ↩

[karlicoss]

So I don't think it's specific to just macos, I installed in a Docker image, and DESTDIRS also contains multiple space separated paths (so easy_install would have the same issue)

FROM ubuntu:23.10

RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y \
        make \
        build-essential \
        libssl-dev \
        zlib1g-dev \
        libbz2-dev \
        libreadline-dev \
        libsqlite3-dev \
        wget \
        curl \
        llvm \
        libncurses5-dev \
        libncursesw5-dev \
        xz-utils \
        tk-dev \
        libffi-dev \
        liblzma-dev \
        git 

RUN git clone https://github.com/pyenv/pyenv.git /pyenv
ENV PYENV_ROOT /pyenv
RUN /pyenv/bin/pyenv install 3.12.0
RUN /pyenv/bin/pyenv global  3.12.0
RUN echo "eval \"\$(/pyenv/bin/pyenv init -)\"" >> /root/.bashrc

and then in the container:

$ python3 --version && python3 -c 'import sysconfig; print({k: v for k, v in sysconfig.get_config_vars().items() if isinstance(v, str) and v.startswith("/pyenv/") and " " in v})'
Python 3.12.0
{'DESTDIRS': '/pyenv/versions/3.12.0 /pyenv/versions/3.12.0/lib /pyenv/versions/3.12.0/lib/python3.12 /pyenv/versions/3.12.0/lib/python3.12/lib-dynload', 'INCLDIRSTOMAKE': '/pyenv/versions/3.12.0/include /pyenv/versions/3.12.0/include /pyenv/versions/3.12.0/include/python3.12 /pyenv/versions/3.12.0/include/python3.12'}

The only issue is I didn't actually manage to reproduce the issue since for some reason pip editable install doesn't invoke easy_install (even though I didn't use --use-pep517 flag). But hopefully it's easy to see from the above output and this code that the spaces would cause a problem

setuptools/setuptools/command/easy_install.py

Lines 1366 to 1374 in 82f5c60

	def create_home_path(self):
	"""Create directories under ~."""
	if not self.user:
	return
	home = convert_path(os.path.expanduser("~"))
	for path in only_strs(self.config_vars.values()):
	if path.startswith(home) and not os.path.isdir(path):
	self.debug_print("os.makedirs('%s', 0o700)" % path)
	os.makedirs(path, 0o700)