Pinned dependency in requirements.txt works, but dependency in another package causes python version check problem

[kwsp]

Description

Pinning dep versions in requirements.txt works as expected in general, but I found some weird behaviour.

Given this requirements.txt:

numba==0.57.0rc1
rocket-fft

Run pip install -r requirements.txt

In a python 3.10 environment, this works perfectly fine. The pinned numba version is installed.

However, in a python 3.11 environment, while trying to setup rocket-fft, pip tries to install the latest stable numba=0.56.4 (maybe because rocket-fft requires numba>=0.56.0), which breaks because numba==0.56 doesn't support python 3.11. (Hence why I pinned the 0.57.0rc1 in the requirements.txt).

Error trace:

$ pip install -r requiements.txt
Collecting numba==0.57.0rc1 (from -r requirements.txt (line 1))
  Using cached numba-0.57.0rc1-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (3.6 MB)
Collecting rocket-fft (from -r requirements.txt (line 2))
  Using cached rocket-fft-0.2.0.tar.gz (42 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [29 lines of output]
      Collecting setuptools>=59.2.0
        Using cached setuptools-67.7.1-py3-none-any.whl (1.1 MB)
      Collecting numba>=0.56.0
        Using cached numba-0.56.4.tar.gz (2.4 MB)
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'error'
        error: subprocess-exited-with-error

        × python setup.py egg_info did not run successfully.
        │ exit code: 1
        ╰─> [8 lines of output]
            Traceback (most recent call last):
              File "<string>", line 2, in <module>
              File "<pip-setuptools-caller>", line 34, in <module>
              File "/tmp/pip-install-cjq2ceft/numba_038204029f80492abe9a98cd3f937572/setup.py", line 51, in <module>
                _guard_py_ver()
              File "/tmp/pip-install-cjq2ceft/numba_038204029f80492abe9a98cd3f937572/setup.py", line 48, in _guard_py_ver
                raise RuntimeError(msg.format(cur_py, min_py, max_py))
            RuntimeError: Cannot install on Python version 3.11.3; only versions >=3.7,<3.11 are supported.
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      × Encountered error while generating package metadata.
      ╰─> See above for output.

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for details.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

This is weird because 0.57.0rc1 should've satisfied rocket-fft's requirement of numba>=0.56.0, but pip still tries to install numba=0.56.4 even when I have numba=0.57.0rc1 installed during the setup of rocket-fft.

Expected behavior

Pinned numba version should've resolved this.

pip version

23.1

Python version

3.11.3

OS

linux

How to Reproduce

Mentioned above

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[kwsp]

I initially suspected this is because pip downloads wheels first, and setup packages without wheels, then installs the wheels, so when it tries to setup rocket-fft, the numba wheels haven't been installed yet. This turns out not to be the cause, because I am able to first run pip install numba==0.57.0rc1 to install numba, but when I run pip install rocket-fft after, I still get this:

$ pip install rocket-fft
Collecting rocket-fft
  Using cached rocket-fft-0.2.0.tar.gz (42 kB)
  Installing build dependencies ... error
  error: subprocess-exited-with-error

  × pip subprocess to install build dependencies did not run successfully.
  │ exit code: 1
  ╰─> [29 lines of output]
      Collecting setuptools>=59.2.0
        Using cached setuptools-67.7.1-py3-none-any.whl (1.1 MB)
      Collecting numba>=0.56.0
        Using cached numba-0.56.4.tar.gz (2.4 MB)
        Preparing metadata (setup.py): started
        Preparing metadata (setup.py): finished with status 'error'
        error: subprocess-exited-with-error

        × python setup.py egg_info did not run successfully.
        │ exit code: 1
        ╰─> [8 lines of output]
            Traceback (most recent call last):
              File "<string>", line 2, in <module>
              File "<pip-setuptools-caller>", line 34, in <module>
              File "/tmp/pip-install-2ruhc46b/numba_d4ac8c3f213d446c9fc3de972f16810c/setup.py", line 51, in <module>
                _guard_py_ver()
              File "/tmp/pip-install-2ruhc46b/numba_d4ac8c3f213d446c9fc3de972f16810c/setup.py", line 48, in _guard_py_ver
                raise RuntimeError(msg.format(cur_py, min_py, max_py))
            RuntimeError: Cannot install on Python version 3.11.3; only versions >=3.7,<3.11 are supported.
            [end of output]

        note: This error originates from a subprocess, and is likely not a problem with pip.
      error: metadata-generation-failed

      × Encountered error while generating package metadata.
      ╰─> See above for output.

      note: This is an issue with the package mentioned above, not pip.
      hint: See above for details.
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

[kwsp]

Do rc candidates not satisfy the >= requirement? I can't find this in the pip documentation anywhere.

Anyhow, I think the == pin should've overridden this requirement.

[groodt]

Edit: I misread the error message. Try with —pre to enable pre releases. https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-pre

What is happening here is that rocket-fft has a build time dependency on numba, and won’t resolve to the pre release of numba (which is aiming to support 3.11).

Original response below:

This is not a pip issue.
note: This error originates from a subprocess, and is likely not a problem with pip.

This is due to rocket-fft and numba not supporting python 3.11.

The projects don’t distribute wheels for 3.11 on PyPI, so it falls back to building from the sdist.

The sdist intentionally raises an exception for python 3.11

https://github.com/styfenschaer/rocket-fft/blob/587ec96d714d804c714239fea1f4da91f44c10e5/setup.py#L12

https://github.com/numba/numba/blob/74a084010755401b001ad2e3ff7b02c154b12b3d/setup.py#L50

I suggest you raise an issue with the upstream projects and ask for 3.11 support, or for them to remove the upper bounds checks in python versions. A warning would likely be more user friendly behaviour.

[pradyunsg]

This is unrelated to rc -- the build failure is due to build isolation.

See https://pip.pypa.io/en/stable/reference/build-system/pyproject-toml/#build-process

Changing the installation in the environment you're running pip from would not affect the isolated build.

[groodt]

What’s the behaviour to enable build time dependencies to resolve pre-release dependencies? Does build isolation need to be disabled?

[pradyunsg]

You'll need to disable build isolation, I believe.

[kwsp]

@pradyunsg build isolation makes sense. If it's currently not possible to enable using pre-releases for deps during build isolation, I see this as a convincing case to add the feature. What if I want to test building my package with RC candidates from a dependency? Part of the point of RC's is to try to build your packages against them and see if they break

[kwsp]

On build isolation - it seems currently if one wants to disable build isolation, one needs to manually install build deps. Would a simple option that allows automatically installing build deps to the current environment (aka just no build isolation be useful for this case)?