[BUG] `pip install -e` broken since setuptools==60.0.0

[ivsanro1]

setuptools version

Since setuptools==60.0.0

Python version

3.7.13

OS

Ubuntu 18.04

Additional environment information

No response

Description

I have a python library in local, which I'm developing. Therefore, I'm interested in installing it with pip install -e as I've always done, to develop it while it's installed.

However, today I updated setuptools from 46.1.3 to the newest (62.1.0) and I realized that even if pip install -e /path/to/my/lib reports no error in output, but I cannot import the library in python.

I've tracked down the version that broke this and it looks like it's 60.0.0.

Install setuptools==59.8.0:

root@7a13ec49fbba:~# pip install --upgrade setuptools==59.8.0
Collecting setuptools==59.8.0
  Using cached setuptools-59.8.0-py3-none-any.whl (952 kB)
Installing collected packages: setuptools
Successfully installed setuptools-59.8.0
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Install library with pip install -e:

root@7a13ec49fbba:~# pip install -e work/ner
Obtaining file:///root/work/ner
Installing collected packages: ner
  Running setup.py develop for ner
Successfully installed ner
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Try to import library in python:

root@7a13ec49fbba:~# python3
Python 3.7.13 (default, Apr 24 2022, 01:04:09) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ner
>>> ner.__version__
'1.0.0'

It works with setuptools==59.8.0.

Now let's do the same with setuptools==60.0.0:

root@7a13ec49fbba:~# pip install --upgrade setuptools==60.0.0
Collecting setuptools==60.0.0
  Downloading setuptools-60.0.0-py3-none-any.whl (952 kB)
Installing collected packages: setuptools
Successfully installed setuptools-60.0.0
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Install library with pip install -e:

root@7a13ec49fbba:~# pip install -e work/ner
Obtaining file:///root/work/ner
Installing collected packages: ner
  Running setup.py develop for ner
Successfully installed ner
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.

Try to import library in python:

root@7a13ec49fbba:~# python3
Python 3.7.13 (default, Apr 24 2022, 01:04:09) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ner
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'ner'

Expected behavior

The installed library via pip install -e should be able to be imported like with setuptools < 60

How to Reproduce

1. Install setuptools>60 via pip install setuptools==60.0.0
2. Download a library to install it in local. e.g. via git clone https://github.com/ivsanro1/gft-ner
3. Try to install library via pip install -e. e.g. pip install -e ~/repos/gft-ner
4. Try to import the library in python, via python3 -c "import ner"

Output

root@7a13ec49fbba:~# pip install setuptools==60.0.0
Collecting setuptools==60.0.0
  Using cached setuptools-60.0.0-py3-none-any.whl (952 kB)
Installing collected packages: setuptools
Successfully installed setuptools-60.0.0
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
root@7a13ec49fbba:~# cd repos/
root@7a13ec49fbba:~/repos# git clone https://github.com/ivsanro1/gft-ner
Cloning into 'gft-ner'...
remote: Enumerating objects: 60, done.
remote: Counting objects: 100% (60/60), done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 60 (delta 21), reused 60 (delta 21), pack-reused 0
Unpacking objects: 100% (60/60), done.
root@7a13ec49fbba:~/repos# cd ..
root@7a13ec49fbba:~# pip install -e repos/gft-ner
Obtaining file:///root/repos/gft-ner
Installing collected packages: ner
  Running setup.py develop for ner
Successfully installed ner
WARNING: You are using pip version 20.2.2; however, version 22.0.4 is available.
You should consider upgrading via the '/usr/bin/python3 -m pip install --upgrade pip' command.
root@7a13ec49fbba:~# python3 -c "import ner"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'ner'

[codeandfire]

I am facing the same issue. Python version is 3.10 and I am running Arch Linux. I can't downgrade to lower versions of setuptools since I am facing different issues with them - the package name comes as UNKNOWN (see this issue). Currently I am managing by manually adding the Python source directory to path. Would really appreciate a fix for this issue. Thanks!

[abravalheri]

Hi guys, thank you for reporting this. Could you try again with the latest version of pip/setuptools?
I did the following test and everything seems to be working fine:

cd /tmp
git clone https://github.com/ivsanro1/gft-ner
cd gft-ner
virtualenv -p py37 .venv
.venv/bin/python -m pip install -U pip setuptools
.venv/bin/python -m pip install -e .
cd ..
gft-ner/.venv/bin/python -c 'import ner; print(ner.__version__)'  # => 1.0.0

[abravalheri]

I also can run this without a virtual environment (in a container):

> docker run --rm -it python:3.7.13-alpine ash

apk add --update git
python -m pip install -U pip setuptools
cd /tmp
git clone https://github.com/ivsanro1/gft-ner
cd gft-ner
pip install -e .
cd ..
python -c 'import ner; print(ner.__version__)'  # => 1.0.0

Here I am purposefully avoiding using the global pip install in a Debian-derived system, because pip might have some problems selecting versions of setuptools there (see pip's issue 6264).

[codeandfire]

The latest version of setuptools 62.3.2 seems to do it. Thanks!

[ivsanro1]

It still fails in my Ubuntu 18.04 environment. Maybe it's the way in which I'm installing Python3?

Here's my Dockerfile:

FROM ubuntu:18.04

RUN apt-get update
RUN apt install -y git

################## INSTALL PYTHON & PIP ##################
RUN apt-get install -y software-properties-common
RUN add-apt-repository ppa:deadsnakes/ppa
RUN apt-get install -y python3.7 python3.7-dev

# If run before install pip, so pip installs for 3.7
RUN update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1
RUN apt-get install -y python3-pip
RUN ln -s /usr/bin/pip3 /usr/bin/pip
##########################################################

RUN python3 -c 'import sys; print(f"\n\n\nPYTHON VERSION: {sys.version}\n\n\n")'

# From here it's basically @abravalheri's steps adapted to the environment
RUN python3 -m pip install -U pip setuptools

WORKDIR /tmp
RUN git clone https://github.com/ivsanro1/gft-ner
WORKDIR /tmp/gft-ner
RUN pip install -e .
WORKDIR /

RUN echo `python3 -c 'import ner; print(f"\n\n\nLIBRARY IS INSTALLED - VERSION: {ner.__version__}\n\n\n")'`

Output (only error):

pypa/setuptools#19 [16/17] RUN echo `python3 -c 'import ner; print(f"\n\n\nLIBRARY IS INSTALLED - VERSION: {ner.__version__}\n\n\n")'`
pypa/setuptools#19 sha256:90ba3264713d1a6c2a43cb2819aeadc3edce8e9a85c7767238c832a740fdb80c
pypa/setuptools#19 0.273 Traceback (most recent call last):
pypa/setuptools#19 0.273   File "<string>", line 1, in <module>
pypa/setuptools#19 0.273 ModuleNotFoundError: No module named 'ner'
pypa/setuptools#19 0.274
pypa/setuptools#19 DONE 0.3s

But still, even if my Python 3 installation is the issue, I did not have this problem with setuptools<60. Which can be checked very easily if you change the line in the Dockerfile, from:

RUN python3 -m pip install -U pip setuptools

to:

RUN python3 -m pip install -U pip "setuptools<60"

The output with setuptools<60:

pypa/setuptools#20 [16/17] RUN echo `python3 -c 'import ner; print(f"\n\n\nLIBRARY IS INSTALLED - VERSION: {ner.__version__}\n\n\n")'`
pypa/setuptools#20 sha256:4ba3cc4faffeddd9175e6a1c21466917b409ec493eda764084b69f0b41983ab7
pypa/setuptools#20 0.269 LIBRARY IS INSTALLED - VERSION: 1.0.0
pypa/setuptools#20 DONE 0.3s

[abravalheri]

Maybe it's the way in which I'm installing Python3?

@ivsanro1, that is a possibility...
To be completely sincere with you I don't know if the workaround you are using for installing pip works 100%. The deadsnakes repository does not seem to provide any deb package for pip... You can however use the ensurepip module (that gets installed with python3.7-venv) to install pip...

I manage to get things working on the selected version of Ubuntu, with the following commands:

> docker run --rm -it ubuntu:18.04 bash

apt-get update
apt-get install -y software-properties-common
add-apt-repository -y ppa:deadsnakes/ppa
apt-get update
apt-get install -y git python3.7 python3.7-dev

apt-get install -y python3.7-venv
# python3.7-venv will provide the 'ensurepip' package as a side effect,
# which in turn is used to install pip
python3.7 -m ensurepip

# In Debian-based systems, there is a well-know issue with pip
# which makes old-versions of setuptools to leak into the build
# environment (see pypa/pip#6264), so let's uninstall and reinstall
# setuptools just in case...
python3.7 -m pip uninstall -y setuptools
python3.7 -m pip install -U setuptools

git clone https://github.com/ivsanro1/gft-ner /tmp/gft-ner
cd /tmp/gft-ner
python3.7 -m pip install -e .
python3.7 -c 'import ner; print(ner.__version__)'

The output is 1.0.0 and python3.7 -m pip list will result in:

python3.7 -m pip list
Package             Version
------------------- ---------------
pip                 22.0.4
PyGObject           3.26.1
python-apt          1.6.5+ubuntu0.7
setuptools          62.3.2
unattended-upgrades 0.1

[ivsanro1]

Thank you @abravalheri . This way it works.
It definitely looks like it's some issue with my installation, because it works any other environment I've tried. I will close the issue.

Thank you for your time

[abravalheri]

@ivsanro1 the Debian patches for Python (also present in Ubuntu and deadsnakes) are very complicated and they heavily customize the installation directories for setuptools... That is why I demonstrated first with a virtual environment and then with Alpine.

I am not 100% sure that I am not forgetting anything in the example above, but just to be on the safe side I always recommend using a virtual environment, even if you are using containers (there is some debate in the community if that is really necessary, but I believe that everyone agrees that using a virtual environment inside a container does not hurt).