Setp issue: Stuck at upgrade.sh

[Quembar]

Hi,
I try to install netbox at an openSUSE 15.2. Follow the installation document at https://netbox.readthedocs.io/en/stable/installation/3-netbox/#install-system-packages I installed all packets from the CentOS list.
The upgrade shows:

 # sudo /opt/netbox/upgrade.sh
Removing old virtual environment...
Creating a new virtual environment at /opt/netbox-2.10.4/venv...
Installing Python system packages (pip3 install wheel)...
Collecting wheel
  Using cached wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.36.2
WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.
You should consider upgrading via the '/opt/netbox-2.10.4/venv/bin/python3 -m pip install --upgrade pip' command.
Installing core dependencies (pip3 install -r requirements.txt)...
Collecting Django==3.1.3
  Using cached Django-3.1.3-py3-none-any.whl (7.8 MB)
Collecting django-cacheops==5.1
  Using cached django_cacheops-5.1-py2.py3-none-any.whl (38 kB)
Collecting django-cors-headers==3.5.0
  Using cached django_cors_headers-3.5.0-py3-none-any.whl (11 kB)
Collecting django-debug-toolbar==3.1.1
  Using cached django_debug_toolbar-3.1.1-py3-none-any.whl (197 kB)
Collecting django-filter==2.4.0
  Using cached django_filter-2.4.0-py3-none-any.whl (73 kB)
Collecting django-mptt==0.11.0
  Using cached django_mptt-0.11.0-py2.py3-none-any.whl (109 kB)
Collecting django-pglocks==1.0.4
  Using cached django-pglocks-1.0.4.tar.gz (4.1 kB)
    ERROR: Command errored out with exit status 1:
     command: /opt/netbox-2.10.4/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-vh5b4cd0/django-pglocks/setup.py'"'"'; __file__='"'"'/tmp/pip-install-vh5b4cd0/django-pglocks/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-8chkgcd6
         cwd: /tmp/pip-install-vh5b4cd0/django-pglocks/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/opt/netbox-2.10.4/venv/lib64/python3.6/site-packages/setuptools/__init__.py", line 15, in <module>
        from six import PY3, string_types
    ModuleNotFoundError: No module named 'six'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.
You should consider upgrading via the '/opt/netbox-2.10.4/venv/bin/python3 -m pip install --upgrade pip' command.

BTW: The message about the PIP version is odd there is the version 21.0.1 installed. And even if I run the update commend for PIP before the upgrade.sh, the message is the same.

But the main error seems to be a version/missing file error:

Collecting django-pglocks==1.0.4
  Using cached django-pglocks-1.0.4.tar.gz (4.1 kB)
    ERROR: Command errored out with exit status 1:

How I can solve this?

[candlerb]

Updating pip won't make any difference, because it will use the version inside the venv, not outside.

As an experiment, please try locally patching upgrade.sh (from #5792) and report back if this fixes the problem:

--- upgrade.sh.orig	2021-02-11 18:34:29.091900888 +0000
+++ upgrade.sh	2021-02-11 18:46:11.481406965 +0000
@@ -30,7 +30,7 @@
 source "${VIRTUALENV}/bin/activate"

 # Install necessary system packages
-COMMAND="pip3 install wheel"
+COMMAND="pip3 install wheel 'pip>=21'"
 echo "Installing Python system packages ($COMMAND)..."
 eval $COMMAND || exit 1

[candlerb]

BTW, I can reproduce your problem in the same way as #5792, modified to

lxc launch images:opensuse/15.2/amd64 nbtest
lxc exec nbtest bash
zypper install -y gcc python3 python3-devel python3-pip libxml2-devel libxslt-devel libffi-devel openssl-devel
mkdir -p /opt/netbox/
cd /opt/netbox/
zypper install -y git
git clone -b master https://github.com/netbox-community/netbox.git .
./upgrade.sh

However the temporary fix I proposed doesn't work, I still get the same error as you:

Collecting django-pglocks==1.0.4
  Using cached django-pglocks-1.0.4.tar.gz (4.1 kB)
    ERROR: Command errored out with exit status 1:
     command: /opt/netbox/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-dqd8mvbl/django-pglocks_91cf0cdfb934484fa388c4ab9685961a/setup.py'"'"'; __file__='"'"'/tmp/pip-install-dqd8mvbl/django-pglocks_91cf0cdfb934484fa388c4ab9685961a/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-bfo4aptb
         cwd: /tmp/pip-install-dqd8mvbl/django-pglocks_91cf0cdfb934484fa388c4ab9685961a/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/opt/netbox/venv/lib64/python3.6/site-packages/setuptools/__init__.py", line 15, in <module>
        from six import PY3, string_types
    ModuleNotFoundError: No module named 'six'

It looks like setuptools is badly broken. If I install six, then it changes to

    ModuleNotFoundError: No module named 'appdirs'

and so on.

I was able to make it work by forcing a particular, known-good version of setuptools:

--- a/upgrade.sh
+++ b/upgrade.sh
@@ -30,7 +30,7 @@ eval $COMMAND || {
 source "${VIRTUALENV}/bin/activate"

 # Install necessary system packages
-COMMAND="pip3 install wheel"
+COMMAND="pip3 install wheel 'pip>=21' 'setuptools==39.0.1'"
 echo "Installing Python system packages ($COMMAND)..."
 eval $COMMAND || exit 1

However, it's up to @jeremystretch to decide on the final fix for CentOS and SUSE users (and/or to diagnose the underlying problem with setuptools).

[candlerb]

Additional info: before running the ./upgrade.sh script for the first time, here are the system packages which are installed:

nbtest:/opt/netbox # pip3 list
Package    Version
---------- -------
appdirs    1.4.3
packaging  20.3
pip        10.0.1
pyparsing  2.2.0
setuptools 40.5.0
six        1.14.0
You are using pip version 10.0.1, however version 21.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

nbtest:/opt/netbox # rpm -qa | grep python3
python3-base-3.6.12-lp152.4.14.1.x86_64
libpython3_6m1_0-3.6.12-lp152.4.14.1.x86_64
python3-pyparsing-2.2.0-lp152.3.3.noarch
python3-packaging-20.3-lp152.1.1.noarch
python3-3.6.12-lp152.4.14.1.x86_64
python3-devel-3.6.12-lp152.4.14.1.x86_64
python3-six-1.14.0-lp152.3.1.noarch
python3-appdirs-1.4.3-lp152.3.2.noarch
python3-setuptools-40.5.0-lp152.3.3.1.noarch
python3-pip-10.0.1-lp152.4.6.1.noarch

After running ./upgrade.sh (which fails as originally shown, ModuleNotFoundError: No module named 'six'):

nbtest:/opt/netbox # . venv/bin/activate
(venv) nbtest:/opt/netbox # pip3 list
Package    Version
---------- -------
pip        20.2.3
setuptools 44.1.1
wheel      0.36.2
WARNING: You are using pip version 20.2.3; however, version 21.0.1 is available.
You should consider upgrading via the '/opt/netbox/venv/bin/python3 -m pip install --upgrade pip' command.

Something very ugly is going on. A temporary fix is to pin just the setuptools version to an older one:

--- a/upgrade.sh
+++ b/upgrade.sh
@@ -30,7 +30,7 @@ eval $COMMAND || {
 source "${VIRTUALENV}/bin/activate"

 # Install necessary system packages
-COMMAND="pip3 install wheel"
+COMMAND="pip3 install wheel setuptools==39.0.1"
 echo "Installing Python system packages ($COMMAND)..."
 eval $COMMAND || exit 1

However, once this has been done once, even if you revert upgrade.sh to how it was originally, on next attempt it works again! rm -rf venv also makes no difference. But rm -rf ~/.cache/pip makes the problem reappear.

Python packaging seems badly broken :-(

[candlerb]

I've just tried the solution from #5776, using new version of upgrade.sh

Unfortunately this doesn't fix the issue given here with SUSE:

nbtest:/opt/netbox # ./upgrade.sh
Creating a new virtual environment at /opt/netbox/venv...
Updating pip (pip install --upgrade pip)...
Collecting pip
  Downloading pip-21.0.1-py3-none-any.whl (1.5 MB)
     |████████████████████████████████| 1.5 MB 11.2 MB/s
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-21.0.1
pip 21.0.1 from /opt/netbox/venv/lib64/python3.6/site-packages/pip (python 3.6)
Installing Python system packages (pip install wheel)...
Collecting wheel
  Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Installing collected packages: wheel
Successfully installed wheel-0.36.2
Installing core dependencies (pip install -r requirements.txt)...
Collecting Django==3.1.3
...
Collecting django-pglocks==1.0.4
  Downloading django-pglocks-1.0.4.tar.gz (4.1 kB)
    ERROR: Command errored out with exit status 1:
     command: /opt/netbox/venv/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-eu6_fc7g/django-pglocks_0686b29e31e644c4808cdcdf5d475a60/setup.py'"'"'; __file__='"'"'/tmp/pip-install-eu6_fc7g/django-pglocks_0686b29e31e644c4808cdcdf5d475a60/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-uu_ilk16
         cwd: /tmp/pip-install-eu6_fc7g/django-pglocks_0686b29e31e644c4808cdcdf5d475a60/
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/opt/netbox/venv/lib64/python3.6/site-packages/setuptools/__init__.py", line 15, in <module>
        from six import PY3, string_types
    ModuleNotFoundError: No module named 'six'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/59/76/2c7d104151316077dcaa103e1d5652ded6c32798c8d706570fb7b48385cb/django-pglocks-1.0.4.tar.gz#sha256=3c47c66fbfbd268ef46269673a0516a039539b0972b8ed2ec9cfee44c4b65523 (from https://pypi.org/simple/django-pglocks/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Could not find a version that satisfies the requirement django-pglocks==1.0.4
ERROR: No matching distribution found for django-pglocks==1.0.4
nbtest:/opt/netbox # 

At this point, the following packages are installed:

nbtest:/opt/netbox # . venv/bin/activate
(venv) nbtest:/opt/netbox # pip list
Package    Version
---------- -------
pip        21.0.1
setuptools 44.1.1
wheel      0.36.2
(venv) nbtest:/opt/netbox #

[Quembar]

I use the same versions:

/opt/netbox # . venv/bin/activate
(venv) /opt/netbox # pip list
Package    Version
---------- -------
pip        21.0.1
setuptools 44.1.1
wheel      0.36.2
(venv) /opt/netbox #

and get errors, too.

[candlerb]

I found
pypa/setuptools#964
which brings me to
https://python-packaging-user-guide.readthedocs.io/tutorials/installing-packages/#ensure-pip-setuptools-and-wheel-are-up-to-date

Following the instructions there, and further modifying the new upgrade.sh as follows:

--- upgrade.sh.orig	2021-02-12 14:56:01.992398949 +0000
+++ upgrade.sh	2021-02-12 14:56:49.396667022 +0000
@@ -30,16 +30,11 @@
 source "${VIRTUALENV}/bin/activate"

 # Upgrade pip
-COMMAND="pip install --upgrade pip"
+COMMAND="pip install --upgrade pip setuptools wheel"
 echo "Updating pip ($COMMAND)..."
 eval $COMMAND || exit 1
 pip -V

-# Install necessary system packages
-COMMAND="pip install wheel"
-echo "Installing Python system packages ($COMMAND)..."
-eval $COMMAND || exit 1
-
 # Install required Python packages
 COMMAND="pip install -r requirements.txt"
 echo "Installing core dependencies ($COMMAND)..."

and now it works.

[TheDirkMelvin]

Had the same issue upgrading from 2.10.2 to 2.10.4.
Confirmed the above fix also works for Ubuntu 18.04.5

It has been 2 weeks since 2.10.4 was released....why hasn't the upgrade script been corrected yet?

[candlerb]

I didn't see this problem on Ubuntu 18.04 when I updated to Netbox 2.10.4.

Anyway, see #5776.