Calling zipapp virtualenv with --seeder=pip fails with OSError: [Errno 13] Permission denied: 'RECORD'

[lada-gagina]

Issue

When calling zipapp version of virtualenv with --seeder=pip to download the latest packaging tools from PyPi, it works fine on Ubuntu, but fails on macOS with a runtime error:

ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'RECORD'
Consider using the `--user` option or check the permissions.

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/Users/lada.gagina/Developer/intellij/out/classes/production/intellij.python.helpers/virtualenv.pyz/__main__.py", line 168, in <module>
  File "/Users/lada.gagina/Developer/intellij/out/classes/production/intellij.python.helpers/virtualenv.pyz/__main__.py", line 164, in run
  File "/Users/lada.gagina/Developer/intellij/out/classes/production/intellij.python.helpers/virtualenv.pyz/virtualenv/__main__.py", line 16, in run
  File "/Users/lada.gagina/Developer/intellij/out/classes/production/intellij.python.helpers/virtualenv.pyz/virtualenv/run/__init__.py", line 28, in cli_run
  File "/Users/lada.gagina/Developer/intellij/out/classes/production/intellij.python.helpers/virtualenv.pyz/virtualenv/run/session.py", line 47, in run
  File "/Users/lada.gagina/Developer/intellij/out/classes/production/intellij.python.helpers/virtualenv.pyz/virtualenv/run/session.py", line 60, in _seed
  File "/Users/lada.gagina/Developer/intellij/out/classes/production/intellij.python.helpers/virtualenv.pyz/virtualenv/seed/embed/pip_invoke.py", line 23, in run
  File "/Users/lada.gagina/Developer/intellij/out/classes/production/intellij.python.helpers/virtualenv.pyz/virtualenv/seed/embed/pip_invoke.py", line 31, in _execute
RuntimeError: failed seed with code 1

Environment

• OS: MacOS 10.15.7. It works fine on Ubuntu 20.4.
• I'm using zipapp version of virtualenv

Output of the virtual environment creation

481 setup logging to NOTSET [DEBUG report:43]
601 find interpreter for spec PythonSpec(path=/Library/Developer/CommandLineTools/usr/bin/python3) [INFO builtin:62]
601 proposed PythonInfo(spec=CPython3.7.3.final.0-64, exe=/Library/Developer/CommandLineTools/usr/bin/python3, platform=darwin, version='3.7.3 (default, Sep  5 2019, 17:14:41) \n[Clang 11.0.0 (clang-1100.0.33.8)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:69]
601 accepted PythonInfo(spec=CPython3.7.3.final.0-64, exe=/Library/Developer/CommandLineTools/usr/bin/python3, platform=darwin, version='3.7.3 (default, Sep  5 2019, 17:14:41) \n[Clang 11.0.0 (clang-1100.0.33.8)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:71]
612 filesystem is not case-sensitive [DEBUG info:28]
652 create virtual environment via CPython3macOsFramework(dest=/Users/lada.gagina/Downloads/venv-test11, clear=False, no_vcs_ignore=False, global=False) [INFO session:52]
652 create folder /Users/lada.gagina/Downloads/venv-test11/bin [DEBUG _sync:25]
652 create folder /Users/lada.gagina/Downloads/venv-test11/lib/python3.7/site-packages [DEBUG _sync:25]
652 write /Users/lada.gagina/Downloads/venv-test11/pyvenv.cfg [DEBUG pyenv_cfg:34]
652 	home = /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
652 	implementation = CPython [DEBUG pyenv_cfg:38]
652 	version_info = 3.7.3.final.0 [DEBUG pyenv_cfg:38]
652 	virtualenv = 20.4.6 [DEBUG pyenv_cfg:38]
652 	include-system-site-packages = false [DEBUG pyenv_cfg:38]
652 	base-prefix = /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
653 	base-exec-prefix = /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7 [DEBUG pyenv_cfg:38]
653 	base-executable = /Library/Developer/CommandLineTools/usr/bin/python3 [DEBUG pyenv_cfg:38]
653 symlink /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python to /Users/lada.gagina/Downloads/venv-test11/bin/python [DEBUG _sync:44]
653 symlink /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/Python3 to /Users/lada.gagina/Downloads/venv-test11/.Python [DEBUG _sync:44]
654 Attempting to acquire lock 4536973856 on /Users/lada.gagina/Library/Application Support/virtualenv/unzip/20.4.6/_virtualenv.py.lock [DEBUG filelock:270]
654 Lock 4536973856 acquired on /Users/lada.gagina/Library/Application Support/virtualenv/unzip/20.4.6/_virtualenv.py.lock [INFO filelock:274]
655 Attempting to release lock 4536973856 on /Users/lada.gagina/Library/Application Support/virtualenv/unzip/20.4.6/_virtualenv.py.lock [DEBUG filelock:315]
655 Lock 4536973856 released on /Users/lada.gagina/Library/Application Support/virtualenv/unzip/20.4.6/_virtualenv.py.lock [INFO filelock:318]
655 create virtualenv import hook file /Users/lada.gagina/Downloads/venv-test11/lib/python3.7/site-packages/_virtualenv.pth [DEBUG api:95]
655 create /Users/lada.gagina/Downloads/venv-test11/lib/python3.7/site-packages/_virtualenv.py [DEBUG api:98]
656 ============================== target debug ============================== [DEBUG session:54]
656 Attempting to acquire lock 4533826952 on /Users/lada.gagina/Library/Application Support/virtualenv/unzip/20.4.6/debug.py.lock [DEBUG filelock:270]
657 Lock 4533826952 acquired on /Users/lada.gagina/Library/Application Support/virtualenv/unzip/20.4.6/debug.py.lock [INFO filelock:274]
657 debug via /Users/lada.gagina/Downloads/venv-test11/bin/python '/Users/lada.gagina/Library/Application Support/virtualenv/unzip/20.4.6/debug.py' [DEBUG creator:224]
716 Attempting to release lock 4533826952 on /Users/lada.gagina/Library/Application Support/virtualenv/unzip/20.4.6/debug.py.lock [DEBUG filelock:315]
716 Lock 4533826952 released on /Users/lada.gagina/Library/Application Support/virtualenv/unzip/20.4.6/debug.py.lock [INFO filelock:318]
656 {
  "sys": {
    "executable": "/Library/Developer/CommandLineTools/usr/bin/python3",
    "_base_executable": "/Library/Developer/CommandLineTools/usr/bin/python3",
    "prefix": "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7",
    "base_prefix": "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7",
    "real_prefix": null,
    "exec_prefix": "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7",
    "base_exec_prefix": "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7",
    "path": [
      "/Users/lada.gagina/Downloads",
      "/usr/local/lib/python2.7/site-packages",
      "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python37.zip",
      "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7",
      "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/lib-dynload",
      "/Users/lada.gagina/Library/Python/3.7/lib/python/site-packages",
      "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site-packages",
      "/Library/Python/3.7/site-packages"
    ],
    "meta_path": [
      "<class '_frozen_importlib.BuiltinImporter'>",
      "<class '_frozen_importlib.FrozenImporter'>",
      "<class '_frozen_importlib_external.PathFinder'>"
    ],
    "fs_encoding": "utf-8",
    "io_encoding": "UTF-8"
  },
  "version": "3.7.3 (default, Sep  5 2019, 17:14:41) \n[Clang 11.0.0 (clang-1100.0.33.8)]",
  "makefile_filename": "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin/Makefile",
  "os": "<module 'os' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/os.py'>",
  "site": "<module 'site' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/site.py'>",
  "datetime": "<module 'datetime' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/datetime.py'>",
  "math": "<module 'math' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/lib-dynload/math.cpython-37m-darwin.so'>",
  "json": "<module 'json' from '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/json/__init__.py'>"
} [DEBUG session:55]
716 add seed packages via PipInvoke(download=False, pip=bundle, setuptools=bundle, wheel=bundle) [INFO session:59]
717 Attempting to acquire lock 4533826056 on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/pip-21.1.1-py3-none-any.whl.lock [DEBUG filelock:270]
717 Lock 4533826056 acquired on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/pip-21.1.1-py3-none-any.whl.lock [INFO filelock:274]
717 Attempting to release lock 4533826056 on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/pip-21.1.1-py3-none-any.whl.lock [DEBUG filelock:315]
717 Lock 4533826056 released on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/pip-21.1.1-py3-none-any.whl.lock [INFO filelock:318]
718 got embed update of distribution pip from /Users/lada.gagina/Library/Application Support/virtualenv/wheel/3.7/embed/1/pip.json [DEBUG via_disk_folder:135]
727 got embed update of distribution pip from /Users/lada.gagina/Library/Application Support/virtualenv/wheel/3.7/embed/1/pip.json [DEBUG via_disk_folder:135]
728 Attempting to acquire lock 4534840680 on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/setuptools-56.0.0-py3-none-any.whl.lock [DEBUG filelock:270]
728 Lock 4534840680 acquired on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/setuptools-56.0.0-py3-none-any.whl.lock [INFO filelock:274]
728 Attempting to release lock 4534840680 on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/setuptools-56.0.0-py3-none-any.whl.lock [DEBUG filelock:315]
728 Lock 4534840680 released on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/setuptools-56.0.0-py3-none-any.whl.lock [INFO filelock:318]
729 got embed update of distribution setuptools from /Users/lada.gagina/Library/Application Support/virtualenv/wheel/3.7/embed/1/setuptools.json [DEBUG via_disk_folder:135]
729 got embed update of distribution setuptools from /Users/lada.gagina/Library/Application Support/virtualenv/wheel/3.7/embed/1/setuptools.json [DEBUG via_disk_folder:135]
730 Attempting to acquire lock 4534840512 on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/wheel-0.36.2-py2.py3-none-any.whl.lock [DEBUG filelock:270]
730 Lock 4534840512 acquired on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/wheel-0.36.2-py2.py3-none-any.whl.lock [INFO filelock:274]
730 Attempting to release lock 4534840512 on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/wheel-0.36.2-py2.py3-none-any.whl.lock [DEBUG filelock:315]
730 Lock 4534840512 released on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/wheel-0.36.2-py2.py3-none-any.whl.lock [INFO filelock:318]
731 got embed update of distribution wheel from /Users/lada.gagina/Library/Application Support/virtualenv/wheel/3.7/embed/1/wheel.json [DEBUG via_disk_folder:135]
732 got embed update of distribution wheel from /Users/lada.gagina/Library/Application Support/virtualenv/wheel/3.7/embed/1/wheel.json [DEBUG via_disk_folder:135]
732 Attempting to acquire lock 4536611728 on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/pip-21.1.1-py3-none-any.whl.lock [DEBUG filelock:270]
732 Lock 4536611728 acquired on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/pip-21.1.1-py3-none-any.whl.lock [INFO filelock:274]
732 Attempting to release lock 4536611728 on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/pip-21.1.1-py3-none-any.whl.lock [DEBUG filelock:315]
732 Lock 4536611728 released on /Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/pip-21.1.1-py3-none-any.whl.lock [INFO filelock:318]
733 got embed update of distribution pip from /Users/lada.gagina/Library/Application Support/virtualenv/wheel/3.7/embed/1/pip.json [DEBUG via_disk_folder:135]
733 pip seed by running: /Users/lada.gagina/Downloads/venv-test11/bin/python -m pip -q install --only-binary :all: --disable-pip-version-check --no-index pip==21.1.1 setuptools==56.0.0 wheel==0.36.2 --find-links '/Users/lada.gagina/Library/Application Support/virtualenv/wheel/house' env of {'LESS_TERMCAP_mb': '\x1b[01;31m', 'TERM_PROGRAM': 'Apple_Terminal', 'LESS_TERMCAP_md': '\x1b[01;38;5;74m', 'COLOR_RED': '\\[\\e[31m\\]', 'LESS_TERMCAP_me': '\x1b[0m', 'SHELL': '/bin/bash', 'TERM': 'xterm-256color', 'CLICOLOR': '1', 'TMPDIR': '/var/folders/r4/x5hzc2kn1wz4jn47t1g75xjr0000gn/T/', 'CONDA_SHLVL': '0', 'COLOR_GREEN': '\\[\\033[32m\\]', 'COLOR_BOLD': '\\[\\e[1m\\]', 'TERM_PROGRAM_VERSION': '433', 'COLOR_BLUE': '\\[\\033[36m\\]', 'OLDPWD': '/Users/lada.gagina/Documents/virtualenv-test', 'LESS_TERMCAP_ue': '\x1b[0m', 'TERM_SESSION_ID': '6CCA1AE8-9AA1-4F47-8704-AC3E81FC07C9', 'COLOR_YELLOW': '\\[\\033[33;1m\\]', 'USER': 'lada.gagina', 'CONDA_EXE': '/Users/lada.gagina/opt/anaconda3/bin/conda', 'SSH_AUTH_SOCK': '/private/tmp/com.apple.launchd.5R3XHTsVUC/Listeners', 'COLOR_DEFAULT': '\\[\\e[0m\\]', '_CE_CONDA': '', 'LESS_TERMCAP_us': '\x1b[04;38;5;146m', 'LSCOLORS': 'ExFxBxDxCxegedabagacad', 'PATH': '/Users/lada.gagina/opt/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public', 'LaunchInstanceID': '4E051D87-1FC0-477C-8721-204F1A6D0981', 'PWD': '/Users/lada.gagina/Downloads', 'EDITOR': 'subl -w', 'XPC_FLAGS': '0x0', '_CE_M': '', 'XPC_SERVICE_NAME': '0', 'SHLVL': '1', 'HOME': '/Users/lada.gagina', 'CONDA_PYTHON_EXE': '/Users/lada.gagina/opt/anaconda3/bin/python', 'PYTHONPATH': '/Users/lada.gagina/Library/Application Support/virtualenv/wheel/house/pip-21.1.1-py3-none-any.whl', 'LOGNAME': 'lada.gagina', 'LESS_TERMCAP_so': '\x1b[38;5;246m', 'LC_CTYPE': 'UTF-8', 'COLOR_WHITE': '\\[\\033[m\\]', 'SECURITYSESSIONID': '186a6', 'LESS_TERMCAP_se': '\x1b[0m', '_': '/usr/bin/python3', '__CF_USER_TEXT_ENCODING': '0x1F5:0x0:0x2', 'SDKROOT': '/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk', 'CPATH': '/usr/local/include', 'LIBRARY_PATH': '/usr/local/lib', '__PYVENV_LAUNCHER__': '/Library/Developer/CommandLineTools/usr/bin/python3', 'PLAT': 'macosx-10.14-x86_64', 'PIP_USE_WHEEL': '1', 'PIP_USER': '0', 'PIP_NO_INPUT': '1'} [DEBUG pip_invoke:27]
ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'RECORD'
Consider using the `--user` option or check the permissions.

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "virtualenv.pyz/__main__.py", line 168, in <module>
  File "virtualenv.pyz/__main__.py", line 164, in run
  File "virtualenv.pyz/virtualenv/__main__.py", line 18, in run
  File "virtualenv.pyz/virtualenv/run/__init__.py", line 32, in cli_run
  File "virtualenv.pyz/virtualenv/run/session.py", line 47, in run
  File "virtualenv.pyz/virtualenv/run/session.py", line 60, in _seed
  File "virtualenv.pyz/virtualenv/seed/embed/pip_invoke.py", line 23, in run
  File "virtualenv.pyz/virtualenv/seed/embed/pip_invoke.py", line 31, in _execute
RuntimeError: failed seed with code 1

[dholth]

I can reproduce. Fails when using the system Python, works when using a Python installed in my home directory. Command works run from the command line, fails when using the env provided by virtualenv.

ERROR: Could not install packages due to an OSError.
Consider using the `--user` option or check the permissions.
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/shutil.py", line 788, in move
    os.rename(src, real_dst)
PermissionError: [Errno 13] Permission denied: '/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/site-packages/wheel-0.33.1.dist-info/' -> '/private/tmp/pip-uninstall-o5z_4l5b'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/private/tmp/v/virtualenv/seed/wheels/embed/pip-21.1.1-py3-none-any.whl/pip/_internal/commands/install.py", line 393, in run
    installed = install_given_reqs(
  File "/private/tmp/v/virtualenv/seed/wheels/embed/pip-21.1.1-py3-none-any.whl/pip/_internal/req/__init__.py", line 70, in install_given_reqs
    uninstalled_pathset = requirement.uninstall(
  File "/private/tmp/v/virtualenv/seed/wheels/embed/pip-21.1.1-py3-none-any.whl/pip/_internal/req/req_install.py", line 653, in uninstall
    uninstalled_pathset.remove(auto_confirm, verbose)
  File "/private/tmp/v/virtualenv/seed/wheels/embed/pip-21.1.1-py3-none-any.whl/pip/_internal/req/req_uninstall.py", line 386, in remove
    moved.stash(path)
  File "/private/tmp/v/virtualenv/seed/wheels/embed/pip-21.1.1-py3-none-any.whl/pip/_internal/req/req_uninstall.py", line 275, in stash
    renames(path, new_path)
  File "/private/tmp/v/virtualenv/seed/wheels/embed/pip-21.1.1-py3-none-any.whl/pip/_internal/utils/misc.py", line 328, in renames
    shutil.move(old, new)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/shutil.py", line 800, in move
    rmtree(src)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/shutil.py", line 715, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/shutil.py", line 672, in _rmtree_safe_fd
    onerror(os.unlink, fullname, sys.exc_info())
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/shutil.py", line 670, in _rmtree_safe_fd
    os.unlink(entry.name, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'RECORD'
Removed build tracker: '/private/tmp/pip-req-tracker-o7z3py7j'

I unzipped virtualenv.pyz and edited ABS_HERE = os.path.abspath("/tmp/virtualenv.pyz") to be able to run /usr/bin/python3 -m pdb __main__.py /tmp/vt34 --seeder=pip

cmd is '/private/tmp/vt34/bin/python -m pip -q install --only-binary :all: --disable-pip-version-check --no-index pip==21.1.1 setuptools==56.0.0 wheel==0.36.2 --find-links /private/tmp/v/virtualenv/seed/wheels/embed'

[pombredanne]

This issue still exists on this setup: stock macOS on ARM/M1

$ python3
Python 3.8.2 (default, Jun  8 2021, 11:59:35) 
[Clang 12.0.5 (clang-1205.0.22.11)] on darwin

$ uname -a
Darwin computer2.local 20.4.0 Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:41 PDT 2021; root:xnu-7195.101.2~1/RELEASE_ARM64_T8101 arm64

[pombredanne]

I am using this command to create clean virtualenvs that do not further fetch or depend anything else:

    wget -O venv/virtualenv.pyz https://bootstrap.pypa.io/virtualenv/virtualenv.pyz
    python3 venv/virtualenv.pyz --never-download --seeder pip  \
      --pip embed --setuptools embed --wheel embed \
      --no-vcs-ignore --no-periodic-update ./venv

This has been working nicely on Linux, macOS x86 and Windows for a long while

[pombredanne]

and removing the seeder with python3 virtualenv.pyz --never-download --pip embed --setuptools embed --wheel embed --no-vcs-ignore --no-periodic-update . works.