Build environment is not isolated when using Apple Python and virtualenv

[Lenderson2]

Thank you for providing feedback on Python packaging!

To help us help you, please fill out as much of the following as you can. If a question is not relevant, feel free to skip it.

1. What is your operating system and version?

macOS 10.15.7

2. What is your Python version?

python --version -> 2.7.16
python3 -- version -> 3.7.3

3. What version of pip do you have?

python -m pip --version
-> /System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python: No module named pip

python3 -m pip --version
-> pip 21.1.1 from /Users/leehenderson/Library/Python/3.7/lib/python/site-packages/pip (python 3.7)

4. If following an online tutorial or guide, please provide a link to the page or section giving you trouble:

https://packaging.python.org/tutorials/packaging-projects/

5. Could you describe your issue in as much detail as possible?

• I have followed the tutorial exactly up to the point of running "python3 -m build" in the root directory of my project.
• I immediately get an error that ends with
  subprocess.CalledProcessError: Command '['/var/folders/xy/vvr9444d0w75h5837pb5w82w0000gn/T/build-env-jdtxwbzl/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' died with <Signals.SIGABRT: 6>.

Here is the stack trace:

Lees-MBP:packaging_tutorial leehenderson$ python3 -m build
Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/__main__.py", line 52, in _build_in_isolated_env
    with IsolatedEnvBuilder() as env:
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/env.py", line 91, in __enter__
    executable, scripts_dir = _create_isolated_env_venv(self._path)
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/env.py", line 199, in _create_isolated_env_venv
    venv.EnvBuilder(with_pip=True).create(path)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/venv/__init__.py", line 68, in create
    self._setup_pip(context)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/venv/__init__.py", line 261, in _setup_pip
    subprocess.check_output(cmd, stderr=subprocess.STDOUT)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/subprocess.py", line 395, in check_output
    **kwargs).stdout
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/subprocess.py", line 487, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/var/folders/xy/vvr9444d0w75h5837pb5w82w0000gn/T/build-env-wxoej6zi/bin/python3', '-Im', 'ensurepip', '--upgrade', '--default-pip']' died with <Signals.SIGABRT: 6>.

I'm new to building packages, so your help would be greatly appreciated! Thank you!

[Lenderson2]

I borrowed this bit python3 -m pip install -U build[virtualenv] from pypa/packaging-problems#505 (comment) and I think that unblocked me for the moment. Now dealing with a permissions error when build tries to satisfy the setuptools>=42 requirement.

Collecting setuptools>=42 (from -r /var/folders/xy/vvr9444d0w75h5837pb5w82w0000gn/T/build-reqs-fe2rv3x5.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d0/15/5041473f5d142ee93bf1593deb8f932e27a078f6f04e2020cf44044f72c5/setuptools-56.2.0-py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 40.8.0
    Uninstalling setuptools-40.8.0:
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/__pycache__/easy_install.cpython-37.pyc'
Consider using the `--user` option or check the permissions.

Weirdly though it's also telling me

You are using pip version 19.0.3, however version 21.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

even though

$ pip install --upgrade pip
Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: pip in /Users/leehenderson/Library/Python/3.7/lib/python/site-packages (21.1.1)

Seems like I might have another version of pip confusingly floating around on my machine?

Full stack trace:

$ python3 -m build
Requirement already satisfied: wheel in /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages (from -r /var/folders/xy/vvr9444d0w75h5837pb5w82w0000gn/T/build-reqs-fe2rv3x5.txt (line 1)) (0.33.1)
Collecting setuptools>=42 (from -r /var/folders/xy/vvr9444d0w75h5837pb5w82w0000gn/T/build-reqs-fe2rv3x5.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/d0/15/5041473f5d142ee93bf1593deb8f932e27a078f6f04e2020cf44044f72c5/setuptools-56.2.0-py3-none-any.whl
Installing collected packages: setuptools
  Found existing installation: setuptools 40.8.0
    Uninstalling setuptools-40.8.0:
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages/__pycache__/easy_install.cpython-37.pyc'
Consider using the `--user` option or check the permissions.

You are using pip version 19.0.3, however version 21.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/__main__.py", line 214, in <module>
    main(sys.argv[1:], 'python -m build')
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/__main__.py", line 206, in main
    build_package(args.srcdir, outdir, distributions, config_settings, not args.no_isolation, args.skip_dependencies)
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/__main__.py", line 94, in build_package
    _build_in_isolated_env(builder, outdir, distributions, config_settings)
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/__main__.py", line 56, in _build_in_isolated_env
    env.install(builder.build_dependencies)
  File "/Users/leehenderson/Library/Python/3.7/lib/python/site-packages/build/env.py", line 168, in install
    subprocess.check_call(cmd)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/subprocess.py", line 347, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/private/var/folders/xy/vvr9444d0w75h5837pb5w82w0000gn/T/build-env-t6hs_b0l/bin/python', '-Im', 'pip', 'install', '--use-pep517', '--no-warn-script-location', '-r', '/var/folders/xy/vvr9444d0w75h5837pb5w82w0000gn/T/build-reqs-fe2rv3x5.txt']' returned non-zero exit status 1.

[Lenderson2]

sudo python3 -m build worked now, but I'm left feeling kind of confused.

Can someone help me understand why I was required to sudo the command when the tutorial didn't need elevated permissions?

And why is my traceback saying You are using pip version 19.0.3, however version 21.1.1 is available. when python3 -m pip --version says I have version 21.1.1?

I know these questions are not directly related to packaging, but I would very much appreciate the help understanding what's going on. Many thanks :)

[layday]

Python comes bundled with an older version of pip which gets installed in a fresh virtual environment such as the one created by build. The version of pip that shipped with Python 3.7.3 was 19.0.3. You can disregard the message about pip being out of date.

It looks like the virtual environment from https://github.com/pypa/packaging-problems/issues/511#issuecomment-840886287 did not actually isolate the system packages. It should not have been able to locate wheel and it should not have attempted to uninstall setuptools. If sudo worked, it's probably because it succeeded at reinstalling setuptools.

[layday]

Actually, virtualenv has a newer version of pip, so if its pip were being used, that message would not have been emitted. There's something going on with the Python path here that's throwing build off.

[layday]

I tried this with the Command Line Tools copy of Python and can confirm that a virtual environment created with virtualenv inherits the path of the caller - the virtualenv Python ends up calling the pip which is visible to build. Observe:

$ virtualenv/bin/python -m inspect -d pip
Target: pip
Origin: /[...]/virtualenv/lib/python3.8/site-packages/pip/__init__.py
Cached: /Users/layday/Library/Caches/com.apple.python/[...]/virtualenv/lib/python3.8/site-packages/pip/__init__.cpython-38.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x102a6b580>
Submodule search path: ['/[...]/virtualenv/lib/python3.8/site-packages/pip']

$ python3 -c "import subprocess
  print(subprocess.check_output(['virtualenv/bin/python', '-m', 'inspect', '-d', 'pip']).decode())"
Target: pip
Origin: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip/__init__.py
Cached: /Users/layday/Library/Caches/com.apple.python/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip/__init__.cpython-38.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x1105122e0>
Submodule search path: ['/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip']

This does not happen with a virtual environment created with the venv module:

$ python3 -c "import subprocess
  print(subprocess.check_output(['venv/bin/python', '-m', 'inspect', '-d', 'pip']).decode())"
Target: pip
Origin: /[...]/venv/lib/python3.8/site-packages/pip/__init__.py
Cached: /Users/layday/Library/Caches/com.apple.python/[...]/venv/lib/python3.8/site-packages/pip/__init__.cpython-38.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x107dfc220>
Submodule search path: ['/[...]/venv/lib/python3.8/site-packages/pip']

To reproduce:

$ python3 -m venv venv
$ venv/bin/python3 -m pip -q install virtualenv
WARNING: You are using pip version 19.2.3, however version 21.1.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ venv/bin/virtualenv --version
virtualenv 20.4.6 from /[...]/venv/lib/python3.8/site-packages/virtualenv/__init__.py
$ venv/bin/virtualenv virtualenv
created virtual environment CPython3.8.2.final.0-64 in 507ms
  creator CPython3macOsFramework(dest=/[...]/virtualenv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/Users/layday/Library/Application Support/virtualenv)
    added seed packages: pip==21.1.1, setuptools==56.0.0, wheel==0.36.2
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator
$ python3 -c "import subprocess
  print(subprocess.check_output(['virtualenv/bin/python', '-m', 'inspect', '-d', 'pip']).decode())"
Target: pip
Origin: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip/__init__.py
Cached: /Users/layday/Library/Caches/com.apple.python/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip/__init__.cpython-38.pyc
Loader: <_frozen_importlib_external.SourceFileLoader object at 0x104bbd2e0>
Submodule search path: ['/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/pip']

[layday]

To get your project to build with isolation, uninstall both build and virtualenv and reinstall build from source: python3 -m pip uninstall -y build virtualenv && python3 -m pip install git+https://github.com/pypa/build. Hopefully there should be a new release of build soon which fixes venv creation on macOS with Python from Xcode.

[henryiii]

But is this still an unsolved bug with the virtualenv creation?

[layday]

This is unrelated to #271 if that's what you mean.

[Lenderson2]

@layday thanks for doing the troubleshooting and giving me some solutions. Really appreciate it!