virtualbox vboxfs buggy filesystem causing issues with pip

[binary1230]

Updates: This failure in pip appears to be an issue in the underlying virtualbox vboxfs shared directory not updating correctly between guest and host. I've updated the bug report here https://www.virtualbox.org/ticket/8761

Read below for the original thread trying to figure out what's going on here.

                  Function: pip.installed
                    Result: False
                   Comment: Unable to install from VCS checkout/home/vagrant/reggie-formula/reggie_install/plugins/uber. Error: Obtaining file:///home/vagrant/reggie-formula/reggie_install/plugins/uber
                            Collecting cherrypy==17.3.0 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/0c/20/e239c3afdeed57203cef5a5bf326234d9e17b449023ea504cd8ef1555833/CherryPy-17.3.0-py2.py3-none-any.whl (421kB)
                            Collecting celery==4.1.1 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/99/fa/4049b26bfe71992ecf979acd39b87e55b493608613054089d975418015b7/celery-4.1.1-py2.py3-none-any.whl (394kB)
                            Collecting python-dateutil==2.6.0 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/40/8b/275015d7a9ec293cf1bbf55433258fbc9d0711890a7f6dc538bac7b86bce/python_dateutil-2.6.0-py2.py3-none-any.whl (194kB)
                            Collecting psycopg2==2.7.3.2 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/29/dc/bf65b06574324222e1a9fd2c80a3acdc0b77d5812ac3d0568dede83d6a50/psycopg2-2.7.3.2-cp36-cp36m-manylinux1_x86_64.whl (2.7MB)
                            Collecting py3k-bcrypt==0.3 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/17/15/c2266d9b316324b694628d5b0c6bb90b398b7d2a48f1d802dc70c0742e12/py3k-bcrypt-0.3.tar.gz
                            Collecting stripe==1.25.0 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/88/a2/b7e285476db3a619e39be934f2c0b50398b300e7dad556f09a24915429b1/stripe-1.25.0.tar.gz (175kB)
                            Collecting pytz==2017.2 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/55/62/e7cd0e15b76062d298413f14bb4ec3cd8568a22d274427f9c3c7286969f4/pytz-2017.2-py2.py3-none-any.whl (484kB)
                            Collecting alembic==0.9.1 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/97/00/3e6797a2e4209db69d23b223ae3148d5f3605dafba6a332670de7a12c147/alembic-0.9.1.tar.gz (999kB)
                            Collecting treepoem==1.0.1 (from uber==2017.7)
                     
         Downloading https://files.pythonhosted.org/packages/da/37/df95940f2e9918bfebb1ffd8f26b3aa31ba3ec53c8a56d6a9e7a0cfefffb/treepoem-1.0.1-py2.py3-none-any.whl (299kB)
                            Collecting email_validator==1.0.2 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/7c/19/1715c8e1e21ee82f6a7ceedee666e43dac628929546fb642853c4ce60fb9/email_validator-1.0.2-py2.py3-none-any.whl
                            Collecting phonenumbers==8.8.1 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/10/3d/0c541773a44bf7509e5d4a60c7070498cfd0b2224053a1514357a00f9013/phonenumbers-8.8.1-py2.py3-none-any.whl (2.7MB)
                            Collecting pockets==0.6.2 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/3a/21/8074b659c374036660612106f95c4e61a4ea0d016154c5f303dc825d861c/pockets-0.6.2-py2.py3-none-any.whl
                            Collecting residue==0.2.8 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/87/11/2bab86f878b2aa74c03ba7e1ca3c098adeea6b20b8ec3f9b3786244c7fc7/residue-0.2.8-py2.py3-none-any.whl
                            Collecting XlsxWriter==1.0.2 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/c7/86/748cb5f6ef5ff2d95a7f189ef1c5124f9badc1d1293dbc214c128595e57e/XlsxWriter-1.0.2-py2.py3-none-any.whl (139kB)
                            Collecting uszipcode==0.1.3 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/4f/03/05f350f5f2a8bd5bfc3b508de96ab6fc6ae5a11c4c2440f5734da1114926/uszipcode-0.1.3.zip (4.5MB)
                            Collecting geopy==1.11.0 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/e0/fd/d1b3225959939907a1a87c3623a021742b3b9ef5db35c15cb916e18de42c/geopy-1.11.0-py2.py3-none-any.whl (66kB)
                            Collecting twilio==6.10.0 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/0c/f9/b21feaf508ee65bad7a6827bc1a00439dcea4780c6d2c966980711e547f4/twilio-6.10.0-py2.py3-none-any.whl (760kB)
                            Collecting cherrys==0.4 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/09/d4/e627b90de90e9fcaa9af0309be50a756f310e1a3bebd0e729c0849e78c3d/cherrys-0.4.tar.gz
                            Collecting redis==2.10.6 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/3b/f6/7a76333cf0b9251ecf49efff635015171843d9b977e4ffcf59f9c4428052/redis-2.10.6-py2.py3-none-any.whl (64kB)
                            Collecting ics==0.4 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/c0/92/77d5c49943a670e7721410076c2ffe2d0821140b95cdf464137c64ee6858/ics-0.4-py2.py3-none-any.whl
                            Collecting pillow==5.2.0 (from uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/d1/24/f53ff6b61b3d728b90934bddb4f03f8ab584a7f49299bf3bde56e2952612/Pillow-5.2.0-cp36-cp36m-manylinux1_x86_64.whl (2.0MB)
                            Requirement already satisfied: six>=1.11.0 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from cherrypy==17.3.0->uber==2017.7) (1.12.0)
                            Requirement already satisfied: more-itertools in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from cherrypy==17.3.0->uber==2017.7) (7.0.0)
                            Collecting contextlib2 (from cherrypy==17.3.0->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/a2/71/8273a7eeed0aff6a854237ab5453bc9aa67deb49df4832801c21f0ff3782/contextlib2-0.5.5-py2.py3-none-any.whl
                            Requirement already satisfied: zc.lockfile in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from cherrypy==17.3.0->uber==2017.7) (1.4)
                            Requirement already satisfied: cheroot>=6.2.4 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from cherrypy==17.3.0->uber==2017.7) (6.5.4)
                            Requirement already satisfied: portend>=2.1.1 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from cherrypy==17.3.0->uber==2017.7) (2.4)
                            Collecting kombu<5.0,>=4.2.0 (from celery==4.1.1->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/b7/af/1914e93314f1b98756d5c5e366193124a0ffaab0e6d0e51e0f6f65fa851d/kombu-4.5.0-py2.py3-none-any.whl (185kB)
                            Collecting billiard<3.6.0,>=3.5.0.2 (from celery==4.1.1->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/8b/b7/c2fe04f2522bb02d044347734eeda3ff5c7a632fa7d0401530a371ba73db/billiard-3.5.0.5.tar.gz (150kB)
                            Requirement already satisfied: requests>=0.8.8 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from stripe==1.25.0->uber==2017.7) (2.21.0)
                            Requirement already satisfied: SQLAlchemy>=0.7.6 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from alembic==0.9.1->uber==2017.7) (1.3.3)
                            Collecting Mako (from alembic==0.9.1->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/a1/bb/f4e5c056e883915c37bb5fb6fab7f00a923c395674f83bfb45c9ecf836b6/Mako-1.0.9.tar.gz (459kB)
                            Collecting python-editor>=0.3 (from alembic==0.9.1->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/c6/d3/201fc3abe391bbae6606e6f1d598c15d367033332bd54352b12f35513717/python_editor-1.0.4-py3-none-any.whl
                            Collecting dnspython>=1.15.0 (from email_validator==1.0.2->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/ec/d3/3aa0e7213ef72b8585747aa0e271a9523e713813b9a20177ebe1e939deb0/dnspython-1.16.0-py2.py3-none-any.whl (188kB)
                            Requirement already satisfied: idna>=2.0.0 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from email_validator==1.0.2->uber==2017.7) (2.8)
                            Requirement already satisfied: backports.functools-lru-cache>=1.3 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from residue==0.2.8->uber==2017.7) (1.5)
                            Collecting PyJWT>=1.4.2 (from twilio==6.10.0->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/87/8b/6a9f14b5f781697e51259d81657e6048fd31a113229cf346880bb7545565/PyJWT-1.7.1-py2.py3-none-any.whl
                            Collecting pysocks; python_version >= "3.0" (from twilio==6.10.0->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/53/12/6bf1d764f128636cef7408e8156b7235b150ea31650d0260969215bb8e7d/PySocks-1.6.8.tar.gz (283kB)
                            Collecting arrow==0.4.2 (from ics==0.4->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/35/59/e1b0eb82d635028c2013fb26ddbc62b1586a9c8cc109f0387f8a3586058c/arrow-0.4.2.tar.gz
                            Requirement already satisfied: setuptools in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from zc.lockfile->cherrypy==17.3.0->uber==2017.7) (41.0.1)
                            Requirement already satisfied: tempora>=1.8 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from portend>=2.1.1->cherrypy==17.3.0->uber==2017.7) (1.14.1)
                            Collecting amqp<3.0,>=2.4.0 (from kombu<5.0,>=4.2.0->celery==4.1.1->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/42/ec/cbbaa8f75be8cbd019afb9d63258e2bdc95242f8c46a54bb90db5fef03bd/amqp-2.4.2-py2.py3-none-any.whl (49kB)
                            Requirement already satisfied: chardet<3.1.0,>=3.0.2 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from requests>=0.8.8->stripe==1.25.0->uber==2017.7) (3.0.4)
                            Requirement already satisfied: certifi>=2017.4.17 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from requests>=0.8.8->stripe==1.25.0->uber==2017.7) (2019.3.9)
                            Requirement already satisfied: urllib3<1.25,>=1.21.1 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from requests>=0.8.8->stripe==1.25.0->uber==2017.7) (1.24.2)
                            Requirement already satisfied: MarkupSafe>=0.9.2 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from Mako->alembic==0.9.1->uber==2017.7) (1.1.1)
                            Requirement already satisfied: jaraco.functools>=1.20 in ./reggie-formula/reggie_install/env/lib/python3.6/site-packages (from tempora>=1.8->portend>=2.1.1->cherrypy==17.3.0->uber==2017.7) (2.0)
                            Collecting vine>=1.1.3 (from amqp<3.0,>=2.4.0->kombu<5.0,>=4.2.0->celery==4.1.1->uber==2017.7)
                              Downloading https://files.pythonhosted.org/packages/7f/60/82c03047396126c8331ceb64da1dc52d4f1317209f32e8fe286d0c07365a/vine-1.3.0-py2.py3-none-any.whl
                            Building wheels for collected packages: py3k-bcrypt, stripe, alembic, uszipcode, cherrys, billiard, Mako, pysocks, arrow
                              Building wheel for py3k-bcrypt (setup.py): started
                              Building wheel for py3k-bcrypt (setup.py): finished with status 'done'
                              Stored in directory: /home/vagrant/.cache/pip/wheels/b8/b9/70/8aef67fd20b18ed2beec032333ad433013e84c2a1f0661adcb
                              Building wheel for stripe (setup.py): started
                              Building wheel for stripe (setup.py): finished with status 'done'
                              Stored in directory: /home/vagrant/.cache/pip/wheels/6a/89/09/ac2bb3ed90c55f10766ac5034fcdecae2755cc443ff1f222b1
                              Building wheel for alembic (setup.py): started
                              Building wheel for alembic (setup.py): finished with status 'done'
                              Stored in directory: /home/vagrant/.cache/pip/wheels/bb/c9/7d/aa5a8ea9e9d08de1297f3e59c31eecaad0e40358c2aaf18e34
                              Building wheel for uszipcode (setup.py): started
                              Building wheel for uszipcode (setup.py): finished with status 'done'
                              Stored in directory: /home/vagrant/.cache/pip/wheels/76/10/9c/60cbc47b0f6340267a621e742478c8f0341bda535e0fcbd07d
                              Building wheel for cherrys (setup.py): started
                              Building wheel for cherrys (setup.py): finished with status 'done'
                              Stored in directory: /home/vagrant/.cache/pip/wheels/4e/12/a9/ebf594d18417a9d842fbfd29c0eade2b1da779ae5e4da1e937
                              Building wheel for billiard (setup.py): started
                              Building wheel for billiard (setup.py): finished with status 'done'
                              Stored in directory: /home/vagrant/.cache/pip/wheels/b8/72/0e/39ecdedc4cfc45b693a623732e40dbd4cff5ea5e11775ee591
                              Building wheel for Mako (setup.py): started
                              Building wheel for Mako (setup.py): finished with status 'done'
                              Stored in directory: /home/vagrant/.cache/pip/wheels/46/23/48/366f0d8b14d436e58ad0aef531b14af8d8beabeb2986704bd5
                              Building wheel for pysocks (setup.py): started
                              Building wheel for pysocks (setup.py): finished with status 'done'
                              Stored in directory: /home/vagrant/.cache/pip/wheels/22/5c/b5/12e0dfdfa85bea67b23628b6425fae715c687e947a45ee3df9
                              Building wheel for arrow (setup.py): started
                              Building wheel for arrow (setup.py): finished with status 'done'
                              Stored in directory: /home/vagrant/.cache/pip/wheels/39/30/d0/ca3b3931053d1bf74e4ebb919dd888b1ee6cdab95c0ab70abf
                            Successfully built py3k-bcrypt stripe alembic uszipcode cherrys billiard Mako pysocks arrow
                            Installing collected packages: contextlib2, cherrypy, vine, amqp, kombu, pytz, billiard, celery, python-dateutil, psycopg2, py3k-bcrypt, stripe, Mako, python-editor, alembic, pillow, treepoem, dnspython, email-validator, phonenumbers, pockets, residue, XlsxWriter, uszipcode, geopy, PyJWT, pysocks, twilio, redis, cherrys, arrow, ics, uber
                              Found existing installation: CherryPy 18.1.1
                                Uninstalling CherryPy-18.1.1:
                                  Successfully uninstalled CherryPy-18.1.1 ERROR: Could not install packages due to an EnvironmentError: [Errno 39] Directory not empty: '/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/~herrypy/lib'

main part of install.log, looks like a conflict with CherryPy versions? more later

particularly, this bit:

                              Found existing installation: CherryPy 18.1.1
                                Uninstalling CherryPy-18.1.1:
                                  Successfully uninstalled CherryPy-18.1.1 ERROR: Could not install packages due to an EnvironmentError: [Errno 39] Directory not empty: '/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/~herrypy/lib
'```

[binary1230]

full install.log
install.log

command run was ./reggie_up.sh stock 2019

[binary1230]

comment from @kitsuta is

it’ll do things like decide it’s impossible to install an old version of pytz over a new version, so when Sideboard installs the latest and then Uber tries to install an older one, it explodes

[kitsuta]

Yup, this is exactly what I was referring to. You can see that even though it says it uninstalled the older version, the install directory isn't empty. Originally I was having this happen with pytz, then Eli and I both had it with pockets. I think it might not matter which package it is in particular. It's not Windows-only either, I use a Mac and Eli uses Linux.

Since you mentioned the file system sharing I've been thinking that's a pretty good starting point to look into.

[binary1230]

was searching around, using this a dumping ground for possible leads:

Not quite the same error, would be interesting to check out:

Something about issues uninstalling if the fs you're uninstalling from and the fs that /tmp is on are different:
https://groups.google.com/forum/#!topic/python-virtualenv/GPXAi9WZKIs

https://community.webfaction.com/questions/2287/upgrading-django-from-104-to-114-in-pinax-07-on-webfaction-produces-oserror

[binary1230]

working on repro steps

after this error happens after a fresh vagrant up, then do:

vagrant ssh
then
pip uninstall cherrypy

then one of these will trigger it:
pip install cherrypy==17.3.0
or
pip install cherrypy==18.1.1

I do still bet it has something to do with doing it on a shared FS, but, at least we have an easy repro

[binary1230]

verbose pip install may indicate that this is something to do with the tmp dir and underlying fs

Created temporary directory: /tmp/pip-ephem-wheel-cache-3896refk
Created temporary directory: /tmp/pip-req-tracker-3lr562gl
Created requirements tracker '/tmp/pip-req-tracker-3lr562gl'
Created temporary directory: /tmp/pip-install-xem_y05t
Collecting cherrypy==17.3.0
  1 location(s) to search for versions of cherrypy:
  * https://pypi.org/simple/cherrypy/
  Getting page https://pypi.org/simple/cherrypy/
  Looking up "https://pypi.org/simple/cherrypy/" in the cache
  Request header has "max_age" as 0, cache bypassed
  Starting new HTTPS connection (1): pypi.org:443
  https://pypi.org:443 "GET /simple/cherrypy/ HTTP/1.1" 304 0
  Analyzing links from page https://pypi.org/simple/cherrypy/
    
	
	[SNIPPED BY DOM, lots of similar looking lines]
	
	
    Found link https://files.pythonhosted.org/packages/aa/0e/4e353c47789ccb50130a44e765dae55b3e85abca01ff21930533ab36afc9/CherryPy-18.1.1-py2.py3-none-any.whl#sha256=3c7b27fd1af4964434d821cd139b8d05c8f7fe4f37ae146fddb3f861d80a1da7 (from https://pypi.org/simple/cherrypy/) (requires-python:>=3.5), version: 18.1.1
    Found link https://files.pythonhosted.org/packages/1f/de/3327bd7168be762180924085fecef2e127d128f1d6157f88cd87fdac2971/CherryPy-18.1.1.tar.gz#sha256=6585c19b5e4faffa3613b5bf02c6a27dcc4c69a30d302aba819639a2af6fa48b (from https://pypi.org/simple/cherrypy/) (requires-python:>=3.5), version: 18.1.1
  Using version 17.3.0 (newest of versions: 17.3.0)
  Created temporary directory: /tmp/pip-unpack-hpv522tm
  Looking up "https://files.pythonhosted.org/packages/0c/20/e239c3afdeed57203cef5a5bf326234d9e17b449023ea504cd8ef1555833/CherryPy-17.3.0-py2.py3-none-any.whl" in the cache
  Current age based on date: 132833
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 132833
  Using cached https://files.pythonhosted.org/packages/0c/20/e239c3afdeed57203cef5a5bf326234d9e17b449023ea504cd8ef1555833/CherryPy-17.3.0-py2.py3-none-any.whl
  Downloading from URL https://files.pythonhosted.org/packages/0c/20/e239c3afdeed57203cef5a5bf326234d9e17b449023ea504cd8ef1555833/CherryPy-17.3.0-py2.py3-none-any.whl#sha256=353074f7774bbd8c5b90d31c7b6fcc0621ce8d09d063e0b1ab5d52b61e76e24a (from https://pypi.org/simple/cherrypy/) (requires-python:>=2.7,!=3.0.*,!=3.1.*,!=3.2.*,!=3.3.*)
  Added cherrypy==17.3.0 from https://files.pythonhosted.org/packages/0c/20/e239c3afdeed57203cef5a5bf326234d9e17b449023ea504cd8ef1555833/CherryPy-17.3.0-py2.py3-none-any.whl#sha256=353074f7774bbd8c5b90d31c7b6fcc0621ce8d09d063e0b1ab5d52b61e76e24a to build tracker '/tmp/pip-req-tracker-3lr562gl'
  Removed cherrypy==17.3.0 from https://files.pythonhosted.org/packages/0c/20/e239c3afdeed57203cef5a5bf326234d9e17b449023ea504cd8ef1555833/CherryPy-17.3.0-py2.py3-none-any.whl#sha256=353074f7774bbd8c5b90d31c7b6fcc0621ce8d09d063e0b1ab5d52b61e76e24a from build tracker '/tmp/pip-req-tracker-3lr562gl'
Requirement already satisfied: six>=1.11.0 in ./env/lib/python3.6/site-packages (from cherrypy==17.3.0) (1.12.0)
Requirement already satisfied: more-itertools in ./env/lib/python3.6/site-packages (from cherrypy==17.3.0) (7.0.0)
Requirement already satisfied: zc.lockfile in ./env/lib/python3.6/site-packages (from cherrypy==17.3.0) (1.4)
Requirement already satisfied: portend>=2.1.1 in ./env/lib/python3.6/site-packages (from cherrypy==17.3.0) (2.4)
Requirement already satisfied: contextlib2 in ./env/lib/python3.6/site-packages (from cherrypy==17.3.0) (0.5.5)
Requirement already satisfied: cheroot>=6.2.4 in ./env/lib/python3.6/site-packages (from cherrypy==17.3.0) (6.5.4)
Requirement already satisfied: setuptools in ./env/lib/python3.6/site-packages (from zc.lockfile->cherrypy==17.3.0) (41.0.1)
Requirement already satisfied: tempora>=1.8 in ./env/lib/python3.6/site-packages (from portend>=2.1.1->cherrypy==17.3.0) (1.14.1)
Requirement already satisfied: backports.functools-lru-cache in ./env/lib/python3.6/site-packages (from cheroot>=6.2.4->cherrypy==17.3.0) (1.5)
Requirement already satisfied: pytz in ./env/lib/python3.6/site-packages (from tempora>=1.8->portend>=2.1.1->cherrypy==17.3.0) (2019.1)
Requirement already satisfied: jaraco.functools>=1.20 in ./env/lib/python3.6/site-packages (from tempora>=1.8->portend>=2.1.1->cherrypy==17.3.0) (2.0)
Installing collected packages: cherrypy
  Found existing installation: CherryPy 18.1.1
    Uninstalling CherryPy-18.1.1:
      Created temporary directory: /tmp/pip-uninstall-0wfskyre
      Removing file or directory /home/vagrant/reggie-formula/reggie_install/env/bin/cherryd
      Created temporary directory: /home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/~herryPy-18.1.1.dist-info
      Removing file or directory /home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/CherryPy-18.1.1.dist-info/
      Created temporary directory: /home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/~.errypy
      Removing file or directory /home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/cherrypy/
      Successfully uninstalled CherryPy-18.1.1

  changing mode of /home/vagrant/reggie-formula/reggie_install/env/bin/cherryd to 777
ERROR: Could not install packages due to an EnvironmentError.
Traceback (most recent call last):
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/shutil.py", line 420, in _rmtree_safe_fd
    os.rmdir(name, dir_fd=topfd)
OSError: [Errno 39] Directory not empty: 'scaffold'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 414, in run
    use_user_site=options.use_user_site,
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_internal/req/__init__.py", line 75, in install_given_reqs
    uninstalled_pathset.commit()
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 444, in commit
    self._moved_paths.commit()
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_internal/req/req_uninstall.py", line 284, in commit
    save_dir.cleanup()
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_internal/utils/temp_dir.py", line 83, in cleanup
    rmtree(self.path)
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_vendor/retrying.py", line 49, in wrapped_f
    return Retrying(*dargs, **dkw).call(f, *args, **kw)
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_vendor/retrying.py", line 212, in call
    raise attempt.get()
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_vendor/retrying.py", line 247, in get
    six.reraise(self.value[0], self.value[1], self.value[2])
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_vendor/six.py", line 693, in reraise
    raise value
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_vendor/retrying.py", line 200, in call
    attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 123, in rmtree
    onerror=rmtree_errorhandler)
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/shutil.py", line 480, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/shutil.py", line 422, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/pip/_internal/utils/misc.py", line 135, in rmtree_errorhandler
    func(path)
OSError: [Errno 39] Directory not empty: '/home/vagrant/reggie-formula/reggie_install/env/lib/python3.6/site-packages/~.errypy/scaffold'
Cleaning up...
Removed build tracker '/tmp/pip-req-tracker-3lr562gl'```

[binary1230]

looks very related
the issue: pypa/pip#6426
the fix: https://github.com/pypa/pip/pull/6427/files

[binary1230]

fascinating.

OK, it seems each time the file is some random file.

I think this isn't a pip bug, it might be a virtualbox+vboxfs bug.

I have a new simpler repro: after everything failed once, I tried to rm -rf on the directory that it was complaining "wasn't empty".

check this out:

(env) vagrant@localhost:~/reggie-formula/reggie_install/env/lib/python3.6/site-packages/~4errypy$ rm -rf __pycache__/
rm: cannot remove '__pycache__/': Directory not empty
(env) vagrant@localhost:~/reggie-formula/reggie_install/env/lib/python3.6/site-packages/~4errypy$ ls -alltr __pycache__/
total 8
drwxrwxrwx 1 vagrant vagrant 4096 Apr 27  2019 ..
drwxrwxrwx 1 vagrant vagrant 4096 Apr 27  2019 .

that pycache dir is CLEARLY EMPTY according to linux.

but on windows, the directory is NOT empty:

this has got to be some virtualbox fs bug then.

[binary1230]

rmdir("__pycache__/") = -1 ENOTEMPTY (Directory not empty)

maybe related to this:
https://www.virtualbox.org/ticket/8761

[binary1230]

yea ok it's definitely some kind of virtualbox vboxfs bug / race condition.

There's a nice repro here:
meteor/meteor#1337 (comment)

#!/bin/bash

set -e -x

rm -rf a0 a1 a2
mkdir -p a0/subdir a1/subdir
echo HI >a0/subdir/file
echo HI >a1/subdir/file
mv a1 a2
mv a0 a1
rm a1/subdir/file
cat a2/subdir/file

run that on vboxfs, it fails. run it on the native fs, it's fine.

[binary1230]

it MIGHT be related to https://coderwall.com/p/mvf0aq/vagrant-auto-update-virtualbox-guest-additions

which we currently have disabled auto-updating, and my versions are mismatched, trying to upgrade all virtualbox versions and see if that helps...

[binary1230]

and..... didn't work, same behavior on vbox 5.2.28 with updated guest editions. yikes.

[binary1230]

ok, calling this a night.

As a workaround to the underlying bug, we could try and fix our dependencies to not uninstall anything (like, there's no reason we need to be installing an old version of cherrypy and then install a newer version).

But, that's a pretty flimsy workaround and we could run into other issues later in a different area if we can't trust the underlying vboxfs.

Some workarounds maybe, all are kinda painful:

1. Move away from virtualbox, try libvirt and something like KVM/QEMU (no idea of the implications)
2. Move away from vagrant altogether (urgh) to something like docker
3. Fix the underlying vboxfs bug (maybe a nightmare and complex)
4. Buy a bunch of vmware licenses?