`--require-hashes` does not correctly handle pinned package with extras

[tonybajan]

What did you want to do?

Install this requirements.txt file with pip 21.0.1 or master in a new virtual environment.

A package is pinned to a non-latest version with an extra (here, requirements[security]) and another dependency requires this package without specifying the extra.

Output

Install fails with:

...
Collecting requests[security]==2.24.0
  Using cached requests-2.24.0-py2.py3-none-any.whl (61 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3==1.25.11
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting requests<3
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    requests<3 from https://files.pythonhosted.org/packages/29/c1/24814557f1d22c56d50280771a17307e6bf87b70727d975fd6b2ce6b014a/requests-2.25.1-py2.py3-none-any.whl#sha256=c210084e36a42ae6b9219e00e48287def368a26d03a048ddad7bfee44f75871e (from requests-extra==1.0.0b3->-r requirements.txt (line 116))

The resolver does not recognise that requests[security]==2.24.0 fulfils requests<3, and tries to collect latest requests. This fails as it is has no pinned hash.

Additional information

Installation succeeds with --use-deprecated=legacy-resolver.

If the requirements file has no hashes, installation succeeds with the new resolver: Collecting requests<3 resolves to latest requests (2.25.1) in the install output, but the pinned version (2.24.0) is what ends up installed.

If all packages are already installed in the environment, pip install succeeds (with Requirement already satisfied) even with the new resolver.

[maljub01]

One workaround for this is to use --no-deps, which is a good idea by itself anyway (See the second warning here).

[cas--]

I think this might be the same issue I have encountered with pinned and hash-checking botframework-connector requirements:

Broken requirements file: requirements.txt

Steps to reproduce:

pip install -U pip pip-tools
pip -V
> pip 21.0.1
echo "botframework-connector==4.12.0" > requirements.in
pip-compile requirements.in --generate-hashes
pip install -r requirements.txt

Actual result:

Collecting adal==1.2.1
  Using cached adal-1.2.1-py2.py3-none-any.whl (52 kB)
Collecting botbuilder-schema==4.12.0
  Using cached botbuilder_schema-4.12.0-py2.py3-none-any.whl (34 kB)
Collecting botframework-connector==4.12.0
  Using cached botframework_connector-4.12.0-py2.py3-none-any.whl (67 kB)
Collecting certifi==2020.12.5
  Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB)
Collecting cffi==1.14.5
  Using cached cffi-1.14.5-cp38-cp38-manylinux1_x86_64.whl (411 kB)
Collecting chardet==3.0.4
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting cryptography==3.3.2
  Using cached cryptography-3.3.2-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
Collecting idna==2.10
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting isodate==0.6.0
  Using cached isodate-0.6.0-py2.py3-none-any.whl (45 kB)
Collecting msal==1.6.0
  Using cached msal-1.6.0-py2.py3-none-any.whl (50 kB)
Collecting msrest==0.6.10
  Using cached msrest-0.6.10-py2.py3-none-any.whl (82 kB)
Collecting oauthlib==3.1.0
  Using cached oauthlib-3.1.0-py2.py3-none-any.whl (147 kB)
Collecting pycparser==2.20
  Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB)
Collecting pyjwt[crypto]==1.5.3
  Using cached PyJWT-1.5.3-py2.py3-none-any.whl (17 kB)
Collecting python-dateutil==2.8.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting requests-oauthlib==1.3.0
  Using cached requests_oauthlib-1.3.0-py2.py3-none-any.whl (23 kB)
Collecting requests==2.23.0
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting urllib3==1.25.11
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Collecting PyJWT>=1.0.0
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    PyJWT>=1.0.0 from https://files.pythonhosted.org/packages/b4/9b/8850f99027ed029af6828199cc87179eaccbbf1f9e6e373e7f0177d32dad/PyJWT-2.0.1-py3-none-any.whl#sha256=b70b15f89dc69b993d8a8d32c299032d5355c82f9b5b7e851d1a6d706dffe847 (from adal==1.2.1->-r requirements.txt (line 7))

Workaround

Either install pip <= 20.3.1, use tonybajan --use-deprecated=legacy-resolver option or use maljub01 --no-deps option:

pip install -r requirements.txt --use-deprecated=legacy-resolver
> Successfully installed adal-1.2.1 botbuilder-schema-4.12.0 botframework-connector-4.12.0 certifi-2020.12.5 cffi-1.14.5 chardet-3.0.4 cryptography-3.3.2 idna-2.10 isodate-0.6.0 msal-1.6.0 msrest-0.6.10 oauthlib-3.1.0 pycparser-2.20 pyjwt-1.5.3 python-dateutil-2.8.1 requests-2.23.0 requests-oauthlib-1.3.0 six-1.15.0 urllib3-1.25.11

pip install -r requirements.txt --no-deps
> Successfully installed adal-1.2.1 botbuilder-schema-4.12.0 botframework-connector-4.12.0 certifi-2020.12.5 cffi-1.14.5 chardet-3.0.4 cryptography-3.3.2 idna-2.10 isodate-0.6.0 msal-1.6.0 msrest-0.6.10 oauthlib-3.1.0 pycparser-2.20 pyjwt-1.5.3 python-dateutil-2.8.1 requests-2.23.0 requests-oauthlib-1.3.0 six-1.15.0 urllib3-1.25.11

Edit: I thought I had posted this as off-topic but after looking at the requirements.txt generated by pip-tools I see that it is specifying extras: pyjwt[crypto]

[di]

If I understand correctly, #9775 did not fix this, but made it possible to fix this?

[uranusjr]

More accurately, the basis of #9775 (#9771) makes it possible to fix this.

[di]

Thanks @uranusjr. This is currently blocking CI and deployment for pypa/warehouse, is there any workaround? If not, could you explain at a high level what I'd need to do to fix this issue in pip?

[uranusjr]

I believe it’s possible to work around this by manually “unpacking” extras, e.g. add a line to also provide hashes to requests in addition to requests[security] (using the example provided by OP).

[di]

I tried to create a failing test for this:

def test_new_resolver_hash_with_extras(script):
    parent_path = create_basic_wheel_for_package(
        script, "parent", "0.1.0", depends=["child"]
    )
    parent_hash = hashlib.sha256(parent_path.read_bytes()).hexdigest()

    child_path = create_basic_wheel_for_package(
        script, "child", "0.1.0", extras={"extra": ["extra"]}
    )
    child_hash = hashlib.sha256(child_path.read_bytes()).hexdigest()

    extra_path = create_basic_wheel_for_package(script, "extra", "0.1.0")
    extra_hash = hashlib.sha256(extra_path.read_bytes()).hexdigest()

    requirements_txt = script.scratch_path / "requirements.txt"
    requirements_txt.write_text(
        """
        child[extra]==0.1.0 --hash=sha256:{child_hash}
        parent==0.1.0 --hash=sha256:{parent_hash}
        extra==0.1.0 --hash=sha256:{extra_hash}
        """.format(
            child_hash=child_hash,
            parent_hash=parent_hash,
            extra_hash=extra_hash,
        ),
    )

    script.pip(
        "install",
        "--no-cache-dir",
        "--no-index",
        "--find-links", script.scratch_path,
        "--requirement", requirements_txt,
    )

    assert_installed(script, parent="0.1.0", child="0.1.0", extra="0.1.0")

but it's passing on the current main branch at e6414d6.

Is it possible that this has already been fixed, or is my test not properly exercising this bug?

[uranusjr]

I think the failure case should be like this:

child==0.1.0 --hash=sha256:{child_hash}
parent==0.1.0 --hash=sha256:{parent_hash}
extra==0.1.0 --hash=sha256:{extra_hash}

and change parent to depend on child[extra].