Build dependencies doesn't use correct pinned version, installs numpy twice during build-time

[xmatthias]

Environment

• pip version: 21.0.1
• Python version: 3.9.0
• OS: linux

Description

Using pyproject.toml build-dependencies installs the latest version of a library, even if the same pip command installs a fixed version.
in very some cases (binary compilation) this can lead to errors like the below when trying to import the dependency.

RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "xxx/.venv/lib/python3.9/site-packages/utils_find_1st/__init__.py", line 3, in <module>
    from .find_1st import find_1st 
ImportError: numpy.core.multiarray failed to import

Expected behavior

Build process should use the pinned version of numpy (1.19.5) instead of the latest version (1.20.0 at time of writing). This way, the installation process will be coherent, and problems like this are not possible.

How to Reproduce

• create new environment
• install numpy and py_find_1st (both with pinned dependencies)

python -m venv .venv
. .venv/bin/activate
pip install -U pip
pip install --no-cache numpy==1.19.5 py_find_1st==1.1.4
python -c "import utils_find_1st"

# To make the above work, upgrade numpy to the latest version (which is the one py_find_1st is compiled against).
pip install -U numpy

Output

$ python -m venv .venv
$ . .venv/bin/activate
$ pip install -U pip
Collecting pip
  Using cached pip-21.0.1-py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.2.3
    Uninstalling pip-20.2.3:
      Successfully uninstalled pip-20.2.3
Successfully installed pip-21.0.1
$ pip install --no-cache numpy==1.19.5 py_find_1st==1.1.4
Collecting numpy==1.19.5
  Downloading numpy-1.19.5-cp39-cp39-manylinux2010_x86_64.whl (14.9 MB)
     |████████████████████████████████| 14.9 MB 10.4 MB/s 
Collecting py_find_1st==1.1.4
  Downloading py_find_1st-1.1.4.tar.gz (8.7 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Building wheels for collected packages: py-find-1st
  Building wheel for py-find-1st (PEP 517) ... done
  Created wheel for py-find-1st: filename=py_find_1st-1.1.4-cp39-cp39-linux_x86_64.whl size=30989 sha256=c1fa1330f733111b2b8edc447bec0c54abf3caf79cd5f386f5cbef310d41885c
  Stored in directory: /tmp/pip-ephem-wheel-cache-94uzfkql/wheels/1e/11/33/aa4db0927a22de4d0edde2a401e1cc1f307bc209d1fdf5b104
Successfully built py-find-1st
Installing collected packages: numpy, py-find-1st
Successfully installed numpy-1.19.5 py-find-1st-1.1.4
$ python -c "import utils_find_1st"
RuntimeError: module compiled against API version 0xe but this version of numpy is 0xd
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/xmatt/development/cryptos/freqtrade_copy/.venv/lib/python3.9/site-packages/utils_find_1st/__init__.py", line 3, in <module>
    from .find_1st import find_1st 
ImportError: numpy.core.multiarray failed to import

In verbose mode, the installation of numpy 1.20.0 can be observed, however, even with "-v", the output is VERY verbose.

....
  changing mode of /tmp/pip-build-env-js9tatya/overlay/bin/f2py3.9 to 755
  Successfully installed numpy-1.20.0 setuptools-52.0.0 wheel-0.36.2
  Removed build tracker: '/tmp/pip-req-tracker-9anxsz9d'
  Installing build dependencies ... done

....

An attached version can be found below (created with pip install --no-cache numpy==1.19.5 py_find_1st==1.1.4 -v &> numpy_install.txt).

numpy_install.txt

[pradyunsg]

Please use numpy's oldest-supported-numpy helper for declaring dependency on numpy in pyproject.toml.

[xmatthias]

i don't think you can point it to how the pyproject.toml is done.

It's pip that's installing numpy twice (1.20.0 for building, and 1.19.5 as final version), so this can also happen with any other package combination in theory.

It works fine if you install numpy FIRST, and then the package depending on numpy, as then pip recognizes that a compatible version is available, and doesn't install it again.

If it wasn't with numpy but with another random package, you couldn't point to "oldest-supported-numpy" either.

The build-dependency is specified as "numpy>=1.13.0" - which allows every numpy > 1.13.0.
Using oldest-supported-numpy might even make it worse, as according to the documentation, that would pin the build-version as numpy==1.13.0 - which would break the install completely.

In short, it's pip that should resolve the build-dependency, detect that it's a dependency that's going to be installed anyway, and install numpy first (using this numpy installation for the build of the other package).

[pfmoore]

pip builds in an isolated environment, so numpy isn't going "to be installed anyway" in the sense that you mean. You can use --no-build-isolation to make pip do the build in the current environment, but that has its own issues (not least, you have to manually install the build dependencies). IMO it's better to correctly tell pip what's needed for the build, and what's needed at runtime, and then it's sorted once and for all. Your particular situation may make that more awkward, in which case you need to explore the trade-offs in choices like disabling build isolation.

[xmatthias]

Even with build-isolation, it's at least downloaded twice (so it's a bug in pip) which wouldn't be necessary.

Both 1.19.5 and 1.20.0 are perfectly valid numpy versions to satisfy the build-dependencies, so if i instruct pip to donwload 1.19.5 - why download 1.20.0 too (and on top of that, cause a potential build-compatibility issue alongside that).

edit:
I think there should be the following behaviour:

• if the build-dependency is specifically pinned (numpy==1.20.0) - then the build-installation should use that dependency, and install whatever is given otherwise in the "regular" environment.
• when it's loosely pinned (numpy>=1.13.0) - it should use as build-dependency what's installed in the same command - and ONLY fall back to the latest version if that dependency is not correctly installed to begin with.

[pradyunsg]

Neither of those suggestions work super cleanly and are actually more difficult to understand and explain than "isolated builds are isolated". As of today, you have 2 options: carefully pin the build dependencies, or tell pip to not do build isolation (i.e. you'll manage the build dependencies in the environment).

Beyond that, I'm not excited by the idea of additional complexity in the dependency resolution process, that makes isolated builds depend on existing environment details -- both of your suggestions require adding additional complexity to the already NP-complete problem of dependency resolution, and that code is already complex enough. And, they're "solutions" operating with incomplete information which will certainly miss certain usecases (eg: custom compiled package that wasn't installed via a wheel).

At the end of the day, pip isn't going to be solving every use case perfectly, and this is one of those imperfect cases at the moment. For now, that means additional work on the user's side, and I'm fine with that because we don't have a good way to have the user communicate the complete complexity of build dependencies to pip.

[paulmueller]

I had the same issue today. Since the release of numpy 1.20.0 yesterday, there is a new dimension to this problem.

For instance, I (mostly my users and CI services) usually install the package dclab with

python3 -m venv env
source env/bin/activate
pip install --upgrade pip wheel
pip install dclab[all]

dclab comes with a few cython extensions that need to be built during installation, which is a perfectly normal use-case. This is not one of those imperfect cases.

Now, the problem is that during installation of dclab, pip downloads numpy 1.20.0 and builds the extensions. But in the environment env, pip installs numpy 1.19.5 (pinned by tensorflow). When I then try to import dclab, I get this error (GH Actions):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/paul/repos/dclab/dclab/__init__.py", line 6, in <module>
    from . import definitions as dfn  # noqa: F401
  File "/home/paul/repos/dclab/dclab/definitions.py", line 4, in <module>
    from .rtdc_dataset.ancillaries import AncillaryFeature
  File "/home/paul/repos/dclab/dclab/rtdc_dataset/__init__.py", line 4, in <module>
    from .check import check_dataset  # noqa: F401
  File "/home/paul/repos/dclab/dclab/rtdc_dataset/check.py", line 10, in <module>
    from .core import RTDCBase
  File "/home/paul/repos/dclab/dclab/rtdc_dataset/core.py", line 12, in <module>
    from ..polygon_filter import PolygonFilter
  File "/home/paul/repos/dclab/dclab/polygon_filter.py", line 8, in <module>
    from .external.skimage.measure import points_in_poly
  File "/home/paul/repos/dclab/dclab/external/__init__.py", line 3, in <module>
    from . import skimage
  File "/home/paul/repos/dclab/dclab/external/skimage/__init__.py", line 2, in <module>
    from . import measure  # noqa: F401
  File "/home/paul/repos/dclab/dclab/external/skimage/measure.py", line 7, in <module>
    from .pnpoly import points_in_poly  # noqa: F401
  File "/home/paul/repos/dclab/dclab/external/skimage/pnpoly.py", line 1, in <module>
    from ._pnpoly import _grid_points_in_poly, _points_in_poly
  File "dclab/external/skimage/_pnpoly.pyx", line 1, in init dclab.external.skimage._pnpoly
    #cython: cdivision=True
ValueError: numpy.ndarray size changed, may indicate binary incompatibility. Expected 88 from C header, got 80 from PyObject

As far as I can see, I have only three choices:

• use oldest-supported-numpy in pyproject.toml (which actually works for dclab)
• install dclab with --no-build-isolation, which I cannot really expect from my users.
• switch from tensorflow to pytorch, which is anyway used more in research as I learned today

The best solution to this problem, as far as I can see, would be for pip to be smart about choosing which version of the build dependency in pyproject.toml to install:

• if there is a numpy already installed in the environment, install the same version to build the extension
• otherwise, if the pip install command already came up with a certain version range for numpy, use the highest available version (e.g. pip install dclab[all] tensorflow would tell pip that tensorflow needs numpy 1.19.5 and so it makes sense to use that when building the extensions)

I know that pinning versions is not good, but tensorflow is doing it apparently, and many people use tensorflow.

[EDIT: found out that oldest-supported-numpy works for me]

[xmatthias]

use oldest-supported-numpy in pyproject.toml

Which Version of numpy does that install?
as far as i could tell from looking at that package, it seemed to use the lowest possible numpy version - so your environment would have 1.19.5, but install-dependencies would be 1.13.x.

While it may not cause a problem in this constellation, it might cause a problem once your environment updates to numpy 1.20.0 (which apparently changed the ndarray size, while prior versions didn't).

[paulmueller]

For me it installs numpy 1.17.3; the oldest-supported-numpy package on PyPI states:

install_requires = 
	
	numpy==1.16.0; python_version=='3.5' and platform_system=='AIX'
	numpy==1.16.0; python_version=='3.6' and platform_system=='AIX'
	numpy==1.16.0; python_version=='3.7' and platform_system=='AIX'
	
	numpy==1.18.5; python_version=='3.5' and platform_machine=='aarch64'
	numpy==1.19.2; python_version=='3.6' and platform_machine=='aarch64'
	numpy==1.19.2; python_version=='3.7' and platform_machine=='aarch64'
	numpy==1.19.2; python_version=='3.8' and platform_machine=='aarch64'
	
	numpy==1.13.3; python_version=='3.5' and platform_machine!='aarch64' and platform_system!='AIX'
	numpy==1.13.3; python_version=='3.6' and platform_machine!='aarch64' and platform_system!='AIX' and platform_python_implementation != 'PyPy'
	numpy==1.14.5; python_version=='3.7' and platform_machine!='aarch64' and platform_system!='AIX' and platform_python_implementation != 'PyPy'
	numpy==1.17.3; python_version=='3.8' and platform_machine!='aarch64' and platform_python_implementation != 'PyPy'
	numpy==1.19.3; python_version=='3.9' and platform_python_implementation != 'PyPy'
	
	numpy==1.19.0; python_version=='3.6' and platform_python_implementation=='PyPy'
	numpy==1.19.0; python_version=='3.7' and platform_python_implementation=='PyPy'
	
	numpy; python_version>='3.10'
	numpy; python_version>='3.8' and platform_python_implementation=='PyPy'

I just checked with a pip install numpy==1.20.0 in my environment. Pip complains about about tensorflow being incompatible with it, but dclab imports and the tests run just fine. I assume that is because of the backwards compatibility (https://pypi.org/project/oldest-supported-numpy/):

The reason to use the oldest available Numpy version as a build-time dependency is because of ABI compatibility. Binaries compiled with old Numpy versions are binary compatible with newer Numpy versions, but not vice versa.