pip editable install fails with pyproject.toml, even with no setup.py import error

[AlanCoding]

Environment

• pip version: 20.2.3
• Python version: 3.8.5
• OS: Fedora 32

Description
A project with both setup.py and pyproject.toml will error when doing an editable install via pip install -e . with the error:

(env) [alancoding@alan-red-hat ansible-runner]$ pip install -e .
Obtaining file:///home/alancoding/repos/ansible-runner
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting python-daemon
  Using cached python_daemon-2.2.4-py2.py3-none-any.whl (35 kB)
Requirement already satisfied: pyyaml in /home/alancoding/repos/awx/env/lib/python3.8/site-packages (from ansible-runner==2.0.0) (5.3.1)
Collecting psutil
  Using cached psutil-5.7.2.tar.gz (460 kB)
Requirement already satisfied: pexpect>=4.5 in /home/alancoding/repos/awx/env/lib/python3.8/site-packages (from ansible-runner==2.0.0) (4.8.0)
Requirement already satisfied: six in /home/alancoding/repos/awx/env/lib/python3.8/site-packages (from ansible-runner==2.0.0) (1.15.0)
Requirement already satisfied: lockfile>=0.10 in /home/alancoding/repos/awx/env/lib/python3.8/site-packages (from python-daemon->ansible-runner==2.0.0) (0.12.2)
Requirement already satisfied: setuptools in /home/alancoding/repos/awx/env/lib/python3.8/site-packages (from python-daemon->ansible-runner==2.0.0) (50.3.0)
Collecting docutils
  Using cached docutils-0.16-py2.py3-none-any.whl (548 kB)
Requirement already satisfied: ptyprocess>=0.5 in /home/alancoding/repos/awx/env/lib/python3.8/site-packages (from pexpect>=4.5->ansible-runner==2.0.0) (0.6.0)
Building wheels for collected packages: psutil
  Building wheel for psutil (setup.py) ... done
  Created wheel for psutil: filename=psutil-5.7.2-cp38-cp38-linux_x86_64.whl size=276719 sha256=8d0a6ef64d2e4eec64a8b781ee2b0258ef7eedc62204919a7ad455a1dd4ff127
  Stored in directory: /home/alancoding/.cache/pip/wheels/91/cf/b0/0c9998060b55ca80ea7a50a8639c3bdc6ba886eeff014bc9ac
Successfully built psutil
Installing collected packages: docutils, python-daemon, psutil, ansible-runner
  Running setup.py develop for ansible-runner
    ERROR: Command errored out with exit status 1:
     command: /home/alancoding/repos/awx/env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/alancoding/repos/ansible-runner/setup.py'"'"'; __file__='"'"'/home/alancoding/repos/ansible-runner/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps
         cwd: /home/alancoding/repos/ansible-runner/
    Complete output (3 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    ModuleNotFoundError: No module named 'setuptools'
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/alancoding/repos/awx/env/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/alancoding/repos/ansible-runner/setup.py'"'"'; __file__='"'"'/home/alancoding/repos/ansible-runner/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.

I cannot find a mention of this type of error anywhere else in the issue queue, and I cannot convince myself this is correct behavior.

The build isolation causes setuptools to not be available, and by all means, I realize that pyproject.toml does not list it. However, the setup.py file does not require it. This error was not from running setup.py, but running the custom pip command

command: /home/alancoding/repos/awx/env/bin/python -c 'import sys, setuptools, tokenize...

The venv at /home/alancoding/repos/awx has setuptools, but somehow with the pip subprocess build isolation, it's not available.

Expected behavior
If running with -e, and the project is split-brained between setup.py and pyproject.toml, then it seems like it shouldn't do build isolation with pyproject.toml to turn around and go run setup.py.

I know that -e isn't supported with PEP 518, and this is only here as legacy behavior. But this looks like a regression that could be addressed by being more consistent about using the legacy behavior.

If I temporarily delete pyproject.toml, then it works like a charm. I think that behavior should be the expectation.

How to Reproduce
If you see any issue in how these projects are structured, then I'm happy to make changes, but I have replicated the error I described with both of:

• https://github.com/ansible/ansible-builder/
• https://github.com/ansible/ansible-runner/

We have pyproject.toml in order to adopt the latest stuff, but we have to include setup.py for compatibility, for RHEL in particular. For ansible/ansible-builder at least, we didn't write setup.py, but generated it from the library dephell, and you can view their template here:

https://github.com/dephell/dephell/blob/657b25a0caac5c61f9b9bf5ec85453b9745a1adb/dephell/converters/setuppy.py#L48

The specific purpose of this template is to produce a compatibility-oriented setup.py script in a project which is otherwise using the new standard.

In either of these projects, the steps to reproduce is pip install -e ., from the project root.

Output

Here's the final part of the verbose output for ansible/ansible-builder:

https://gist.github.com/AlanCoding/dab97a4a84e14911fb1f7c1087e437b2

The skip logs were too numerous to include the full output, so the beginning is truncated.

[duckinator]

I've encountered this problem before. My understanding of it is that it needs setuptools installed before it tries using setup.py + setup.cfg. This means you need to specfiy it as a requirement in the build-system table.

For ansible/ansible-builder, this patch was enough to fix it:

From dc6ad207894e62124cbeb4f4aa1cf046a4a1559d Mon Sep 17 00:00:00 2001
From: Ellen Marie Dash <me@duckie.co>
Date: Sat, 17 Oct 2020 21:07:02 -0400
Subject: [PATCH] Fix editable pip installations.

The build process needs setuptools before reading setup.py, which means
it needs to be specified in pyproject.toml's `build-system` table.
---
 pyproject.toml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/pyproject.toml b/pyproject.toml
index e4666a4..f3c682e 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -46,5 +46,5 @@ exclude = '''
 '''
 
 [build-system]
-requires = ["poetry>=1.0.5"]
+requires = ["setuptools", "poetry>=1.0.5"]
 build-backend = "poetry.masonry.api"
-- 
2.25.4

[duckinator]

@AlanCoding I made PRs fixing the problem:

• Fix editable installations using pip install -e . ansible/ansible-builder#118
• Fix editable installations using pip install -e . ansible/ansible-runner#545

[pradyunsg]

However, the setup.py file does not require it.

It doesn't, but PEP 517 doesn't really have a mechanism for editable installs -- so that entire chunk of pip's code does continue to expect setuptools to be available. 🤷🏽‍♂️

[duckinator]

that entire chunk of pip's code does continue to expect setuptools to be available

@pradyunsg iirc, for editable installations it falls back to the non-PEP517 approach, right? Or is it more nuanced than that?

[uranusjr]

I don’t think there is a fallback at all. PEP 517 configuration is simply ignored entirely for an editable installation.

[duckinator]

In terms of avoiding other people encountering this problem, is there a way to check if the program is in the "split-brained" mode and the error was setuptools missing?

If so, perhaps we could catch that and point people to documentation about this, or something like that?