Can't run the tests (several errors) on Windows

[olivierdalang]

Hi,

Following the README.md, I was able to setup the vagrant VM, but the test commands don't work.

I downloaded the box manually from https://www.djangoproject.com/m/vms/djangocore-box-1.1.box (I have an unstable and slow connection, and stupidly, atlas.hashicorp doesn't support resuming downloads). I run Windows 10 64 bits, Virtualbox 5.0.20.

Disclaimer : I never used vagrant boxes before, so maybe I missed an implicit step.

~$ runtests2.7-mysql

fails with

Please install test dependencies first:
$ pip install -r requirements/py2.txt

Then

~$ pip install -r /django/tests/requirements/py2.txt

fails with

Exception:
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/pip/basecommand.py", line 139, in main
    status = self.run(options, args)
  File "/usr/local/lib/python2.7/dist-packages/pip/commands/install.py", line 240, in run
    for req in parse_requirements(filename, finder=finder, options=options):
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1314, in parse_requirements
    for item in parse_requirements(req_url, finder, comes_from=filename, options=options):
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 1356, in parse_requirements
    req = InstallRequirement.from_line(line, comes_from)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 118, in from_line
    return cls(req, comes_from, url=url)
  File "/usr/local/lib/python2.7/dist-packages/pip/req.py", line 43, in __init__
    req = pkg_resources.Requirement.parse(req)
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 2680, in parse
    reqs = list(parse_requirements(s))
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 2605, in parse_requirements
    line, p, specs = scan_list(VERSION,LINE_END,line,p,(1,2),"version spec")
  File "/usr/local/lib/python2.7/dist-packages/distribute-0.6.35-py2.7.egg/pkg_resources.py", line 2573, in scan_list
    raise ValueError("Expected "+item_name+" in",line,"at",line[p:])
ValueError: ('Expected version spec in', "pylibmc; sys.platform != 'win32'", 'at', "; sys.platform != 'win32'")

Python3.3 tests have another problem :

~$ runtests3.3-mysql

fails with

Traceback (most recent call last):
  File "/django/tests/runtests.py", line 13, in <module>
    from django.apps import apps
  File "/django/django/apps/__init__.py", line 1, in <module>
    from .config import AppConfig
  File "/django/django/apps/config.py", line 6, in <module>
    from django.utils.module_loading import module_has_submodule
  File "/django/django/utils/module_loading.py", line 67, in <module>
    from importlib.util import find_spec as importlib_find
ImportError: cannot import name find_spec

[timgraham]

What version of pip is installed? Our requirements files require pip 8+.

The Python 3.3 failure is expected since master doesn't support Python 3. See https://docs.djangoproject.com/en/dev/faq/install/#what-python-version-can-i-use-with-django.

[olivierdalang]

Oops I just saw I downloaded https://www.djangoproject.com/m/vms/djangocore-box-1.1.box , I misread the link and didn't make the difference between 1.1 and 1.11... Sorry for the noise !

[timgraham]

I think I'll remove that image since it's obsolete. Is there a pointer to it that we could remove?

[olivierdalang]

No I found the link on the mailing list thread.
But is the last version hosted somewhere else than on hashicorp's server ? They don't support resuming downloads (!!) making getting the boxes a challenge form here (I'm in Senegal, with a slow and unreliable connection).

[timgraham]

I don't think so. Maybe you could contact them and ask if there's a reason it's not supported.

[jarshwah]

@olivierdalang you could try using the commands to build the box from scratch which only involves an initial 500mb (or so) to get the base ubuntu box. The provisioner will then download the rest of the software needed.

Alternatively, you can download the fully compiled box directly from hashicorp with the following url:

https://atlas.hashicorp.com/djangoproject/boxes/django-box-1.11/versions/1.11.0/virtualbox.box

[olivierdalang]

Ok it works better with the correct version, but I still run into some errors.

(vm) ~$ runtests35-mysql

returned

ERROR:   py35-mysql: InvocationError: /usr/bin/python3 -m virtualenv --python /usr/bin/python3.5 py35-mysql (see /django/.tox/py35-mysql/log/py35-mysql-0.log)

The log file displays

OSError: [Errno 71] Protocol error: '/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu' -> '/django/.tox/py35-mysql/lib/python3.5/config-3.5m-x86_64-linux-gnu'

I think that's a Windows-specific error, where VirtualBox shared folders don't support symbolic links by default. I thought this should be fixable with this :

(host) "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" setextradata django-box_default_1474373922214_44815 VBoxInternal2/SharedFoldersEnableSymlinksCreate/django 1

But it didn't work.

I had more success by changing the tox working directory so that it's not on the shared folder anymore. In /django/tox.ini, add toxworkdir=/tmp/toxwork.

The annoying thing is that tox.ini is versioned. I didn't find a way to override the toxworkdir variable otherwise. Isn't there a way to set it using something like (vm) ~$ runtests35-mysql --toxworkdir=/tmp/toxwork ?

After this

(vm) ~$ runtests35-mysql

returns

py35-mysql: could not install deps [-rtests/requirements/py3.txt, -rtests/requirements/mysql.txt]; v = InvocationError('/tmp/toxwork/py35-mysql/bin/pip install -rtests/requirements/py3.txt -rtests/requirements/mysql.txt (see /tmp/toxwork/py35-mysql/log/py35-mysql-1.log)', 1)

The log file displays

actionid: py35-mysql
msg: getenv
cmdargs: [local('/tmp/toxwork/py35-mysql/bin/pip'), 'install', '-rtests/requirements/py3.txt', '-rtests/requirements/mysql.txt']
env: {'VIRTUAL_ENV': '/tmp/toxwork/py35-mysql', 'SSH_CONNECTION': '10.0.2.2 53363 10.0.2.15 22', 'PWD': '/home/vagrant', 'HOME': '/home/vagrant', 'PYTHONHASHSEED': '614178105', 'XDG_SESSION_ID': '2', 'SHELL': '/bin/bash', 'LOGNAME': 'vagrant', '_': '/usr/local/bin/tox', 'PATH': '/tmp/toxwork/py35-mysql/bin:/home/vagrant/bin:/home/vagrant/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games', 'LC_ALL': 'en_US.UTF-8', 'TERM': 'cygwin', 'LANG': 'en_US.UTF-8', 'DISPLAY': ':99', 'PYTHONPATH': '/home/vagrant/djangodata/', 'SSH_CLIENT': '10.0.2.2 53363 22', 'MAIL': '/var/mail/vagrant', 'XDG_RUNTIME_DIR': '/run/user/900', 'SSH_TTY': '/dev/pts/0', 'USER': 'vagrant', 'LANGUAGE': 'en_US.UTF-8', 'PYTHONDONTWRITEBYTECODE': '1', 'LESSCLOSE': '/usr/bin/lesspipe %s %s', 'SHLVL': '1', 'LESSOPEN': '| /usr/bin/lesspipe %s', 'LS_COLORS': 'rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:'}

Collecting argon2-cffi>=16.1.0 (from -r tests/requirements/base.txt (line 1))
  Using cached argon2_cffi-16.2.0.tar.gz
Collecting bcrypt (from -r tests/requirements/base.txt (line 2))
  Using cached bcrypt-3.1.1-cp35-cp35m-manylinux1_x86_64.whl
Collecting docutils (from -r tests/requirements/base.txt (line 3))
  Using cached docutils-0.12-py3-none-any.whl
Collecting geoip2 (from -r tests/requirements/base.txt (line 4))
  Using cached geoip2-2.4.0-py2.py3-none-any.whl
Collecting jinja2>=2.7 (from -r tests/requirements/base.txt (line 5))
  Using cached Jinja2-2.8-py2.py3-none-any.whl
Collecting numpy (from -r tests/requirements/base.txt (line 6))
  Downloading numpy-1.11.1-cp35-cp35m-manylinux1_x86_64.whl (15.6MB)
Collecting Pillow (from -r tests/requirements/base.txt (line 7))
  Downloading Pillow-3.3.1-cp35-cp35m-manylinux1_x86_64.whl (5.6MB)
Collecting PyYAML (from -r tests/requirements/base.txt (line 8))
  Downloading PyYAML-3.12.tar.gz (253kB)
Collecting pylibmc (from -r tests/requirements/base.txt (line 10))
  Downloading pylibmc-1.5.1.tar.gz (59kB)
Collecting pytz>dev (from -r tests/requirements/base.txt (line 11))
  Downloading pytz-2016.6.1-py2.py3-none-any.whl (481kB)
Collecting selenium (from -r tests/requirements/base.txt (line 12))
  Downloading selenium-2.53.6-py2.py3-none-any.whl (884kB)
Collecting sqlparse (from -r tests/requirements/base.txt (line 13))
  Downloading sqlparse-0.2.1.tar.gz (53kB)
Collecting tblib (from -r tests/requirements/base.txt (line 14))
  Downloading tblib-1.3.0-py2.py3-none-any.whl
Collecting python3-memcached (from -r tests/requirements/py3.txt (line 2))
  Downloading python3-memcached-1.51.tar.gz
Collecting mysqlclient>=1.3.7 (from -r tests/requirements/mysql.txt (line 2))
  Downloading mysqlclient-1.3.7.tar.gz (79kB)
Collecting six (from argon2-cffi>=16.1.0->-r tests/requirements/base.txt (line 1))
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting cffi>=1.0.0 (from argon2-cffi>=16.1.0->-r tests/requirements/base.txt (line 1))
  Downloading cffi-1.8.3-cp35-cp35m-manylinux1_x86_64.whl (397kB)
Collecting maxminddb>=1.2.1 (from geoip2->-r tests/requirements/base.txt (line 4))
  Downloading maxminddb-1.2.1.tar.gz (1.3MB)
Collecting requests>=2.9 (from geoip2->-r tests/requirements/base.txt (line 4))
  Downloading requests-2.11.1-py2.py3-none-any.whl (514kB)
Collecting MarkupSafe (from jinja2>=2.7->-r tests/requirements/base.txt (line 5))
  Downloading MarkupSafe-0.23.tar.gz
Collecting pycparser (from cffi>=1.0.0->argon2-cffi>=16.1.0->-r tests/requirements/base.txt (line 1))
  Downloading pycparser-2.14.tar.gz (223kB)
Building wheels for collected packages: argon2-cffi, PyYAML, pylibmc, sqlparse, python3-memcached, mysqlclient, maxminddb, MarkupSafe, pycparser
  Running setup.py bdist_wheel for argon2-cffi: started
  Running setup.py bdist_wheel for argon2-cffi: finished with status 'done'
  Stored in directory: /home/vagrant/.cache/pip/wheels/f8/f2/af/213f3e49e3b54c3f2bd4ec2da7ca7cbfd8d0262a0d1db22eaa
  Running setup.py bdist_wheel for PyYAML: started
  Running setup.py bdist_wheel for PyYAML: finished with status 'done'
  Stored in directory: /home/vagrant/.cache/pip/wheels/2c/f7/79/13f3a12cd723892437c0cfbde1230ab4d82947ff7b3839a4fc
  Running setup.py bdist_wheel for pylibmc: started
  Running setup.py bdist_wheel for pylibmc: finished with status 'error'
  Complete output from command /tmp/toxwork/py35-mysql/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zuota6lu/pylibmc/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpbfuzimcopip-wheel- --python-tag cp35:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.5
  creating build/lib.linux-x86_64-3.5/pylibmc
  copying src/pylibmc/__main__.py -> build/lib.linux-x86_64-3.5/pylibmc
  copying src/pylibmc/test.py -> build/lib.linux-x86_64-3.5/pylibmc
  copying src/pylibmc/consts.py -> build/lib.linux-x86_64-3.5/pylibmc
  copying src/pylibmc/pools.py -> build/lib.linux-x86_64-3.5/pylibmc
  copying src/pylibmc/client.py -> build/lib.linux-x86_64-3.5/pylibmc
  copying src/pylibmc/__init__.py -> build/lib.linux-x86_64-3.5/pylibmc
  warning: build_py: byte-compiling is disabled, skipping.

  running build_ext
  building '_pylibmc' extension
  creating build/temp.linux-x86_64-3.5
  creating build/temp.linux-x86_64-3.5/src
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE_ZLIB -I/usr/include/python3.5m -I/tmp/toxwork/py35-mysql/include/python3.5m -c src/_pylibmcmodule.c -o build/temp.linux-x86_64-3.5/src/_pylibmcmodule.o -fno-strict-aliasing -std=c99
  In file included from src/_pylibmcmodule.c:34:0:
  src/_pylibmcmodule.h:42:36: fatal error: libmemcached/memcached.h: No such file or directory
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for pylibmc
  Running setup.py clean for pylibmc
  Running setup.py bdist_wheel for sqlparse: started
  Running setup.py bdist_wheel for sqlparse: finished with status 'done'
  Stored in directory: /home/vagrant/.cache/pip/wheels/e2/82/57/0a192624b6d4faa1d64d2b1f0efae857496b4f35fb7e86aba1
  Running setup.py bdist_wheel for python3-memcached: started
  Running setup.py bdist_wheel for python3-memcached: finished with status 'done'
  Stored in directory: /home/vagrant/.cache/pip/wheels/be/3c/13/413aea983af79f9a4b4c2e737a6c8e7802c9cf0d69a7673b60
  Running setup.py bdist_wheel for mysqlclient: started
  Running setup.py bdist_wheel for mysqlclient: finished with status 'done'
  Stored in directory: /home/vagrant/.cache/pip/wheels/09/64/5e/8f66e5b5ddef002da639a9b16b32ecedd4d56c51e6358e4e0b
  Running setup.py bdist_wheel for maxminddb: started
  Running setup.py bdist_wheel for maxminddb: finished with status 'done'
  Stored in directory: /home/vagrant/.cache/pip/wheels/d3/5f/0b/8744068f80d2b3cf0772b529475a1617fa27d50f7bc818d8c3
  Running setup.py bdist_wheel for MarkupSafe: started
  Running setup.py bdist_wheel for MarkupSafe: finished with status 'done'
  Stored in directory: /home/vagrant/.cache/pip/wheels/a3/fa/dc/0198eed9ad95489b8a4f45d14dd5d2aee3f8984e46862c5748
  Running setup.py bdist_wheel for pycparser: started
  Running setup.py bdist_wheel for pycparser: finished with status 'done'
  Stored in directory: /home/vagrant/.cache/pip/wheels/9b/f4/2e/d03e949a551719a1ffcb659f2c63d8444f4df12e994ce52112
Successfully built argon2-cffi PyYAML sqlparse python3-memcached mysqlclient maxminddb MarkupSafe pycparser
Failed to build pylibmc
Installing collected packages: six, pycparser, cffi, argon2-cffi, bcrypt, docutils, maxminddb, requests, geoip2, MarkupSafe, jinja2, numpy, Pillow, PyYAML, pylibmc, pytz, selenium, sqlparse, tblib, python3-memcached, mysqlclient
  Running setup.py install for pylibmc: started
    Running setup.py install for pylibmc: finished with status 'error'
    Complete output from command /tmp/toxwork/py35-mysql/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zuota6lu/pylibmc/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-l0xc1do4-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/toxwork/py35-mysql/include/site/python3.5/pylibmc:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.5
    creating build/lib.linux-x86_64-3.5/pylibmc
    copying src/pylibmc/__main__.py -> build/lib.linux-x86_64-3.5/pylibmc
    copying src/pylibmc/test.py -> build/lib.linux-x86_64-3.5/pylibmc
    copying src/pylibmc/consts.py -> build/lib.linux-x86_64-3.5/pylibmc
    copying src/pylibmc/pools.py -> build/lib.linux-x86_64-3.5/pylibmc
    copying src/pylibmc/client.py -> build/lib.linux-x86_64-3.5/pylibmc
    copying src/pylibmc/__init__.py -> build/lib.linux-x86_64-3.5/pylibmc
    warning: build_py: byte-compiling is disabled, skipping.

    running build_ext
    building '_pylibmc' extension
    creating build/temp.linux-x86_64-3.5
    creating build/temp.linux-x86_64-3.5/src
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DUSE_ZLIB -I/usr/include/python3.5m -I/tmp/toxwork/py35-mysql/include/python3.5m -c src/_pylibmcmodule.c -o build/temp.linux-x86_64-3.5/src/_pylibmcmodule.o -fno-strict-aliasing -std=c99
    In file included from src/_pylibmcmodule.c:34:0:
    src/_pylibmcmodule.h:42:36: fatal error: libmemcached/memcached.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/tmp/toxwork/py35-mysql/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-zuota6lu/pylibmc/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-l0xc1do4-record/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/toxwork/py35-mysql/include/site/python3.5/pylibmc" failed with error code 1 in /tmp/pip-build-zuota6lu/pylibmc/vagrant@djangobox:~$

Here it looks there are missing dependencies. I didn't investigate further.

[timgraham]

Are you using the version released yesterday? #14 (comment)

[olivierdalang]

No. It looks it's the same error on that other issue, I didn't see it.

I'll test again when I have a chance to download the new box.

[olivierdalang]

Hi,

Testing again with the last version, it now works with the following fixes :

1. enable symlink for the virtual machine (replace django-box_default_1474373922214_44815 by the VM name in VirtualBox) :
   (host) "C:\Program Files\Oracle\VirtualBox\VBoxManage.exe" setextradata django-box_default_1474373922214_44815 VBoxInternal2/SharedFoldersEnableSymlinksCreate/django 1
2. make sure to run vagrant as an administrator

Thanks for the support ! I'll make a PR for the readme to add these instructions.

[olivierdalang]

Well sorry I just thought about it again, I think this VBoxInternal2/SharedFoldersEnableSymlinksCreate/django setting could be set to true in the .box file, right ? If so, the only needed step for a windows user would be to run vagrant as an admin.