Updating remote links with new URLs for PEP508 functionallity

[cam72cam]

What's the problem this feature will solve?
Installing a wheel file from a URL will not install a new wheel if the link changes.

Example:
setup.py

install_requires=[
    "package@http://domain:port/link-0.1.3.whl
]

If you change that link and try to re-install or upgrade the parent package, pip ignores the new requirement.

If there is no way for the url/version to be checked when installing dependencies, the URL support added in PEP 508 seems problematic at best.

Describe the solution you'd like
Installing remote links should always be reinstalled, same as editable links seems to be the simplest method.

Alternative Solutions
We could instead try to guess/determine/specify versions in the requirement string.
Ex:

"package=0.1.3@http://domain:port/link-0.1.3.whl"

which would imply that the link provides version 0.1.3 that we could check against the currently installed package version

[pfmoore]

What's the use case for the link changing? In particular, what's the situation where you'd have:

1. The URL changes
2. The project name and version remain the same
3. The code is different

I'm particularly uncomfortable with assuming that the code could be different even thought the name/version is the same, so I'd like to see a strong justification for why that's the case (the assumption that if the name and version are the same, the built project will be, is pretty pervasive throughout the pip codebase).

[cam72cam]

The problem is the name is the same, but the version changes.

If I change "package@http://domain:port/link-0.1.3.whl" to "package@http://domain:port/link-0.2.5.whl", it won't install the new version of the package. When I re-install or upgrade the current package.

The package@remote notation will never install a newer version of the specified package as it's written right now.

[benoit-pierre]

The problem is PEP 508 direct URLs don't allow for version specifiers (otherwise you'd just be able to update the version, like for other requirements), so the only way pip would know the package version changed is to "prepare" it.

[cam72cam]

Exactly

[benoit-pierre]

Frankly, they really are a poor substitute for dependency links.

[cam72cam]

Something I noticed is that pip install functions a bit different. It looks like it compares the remote URL

pip install http://server:port/path/package-0.1.whl
(installs package-0.1)
pip install http://server:port/path/package-0.1.whl
(already installed message)
pip install http://server:port/path/package-0.2.whl
(installs new version)
pip install http://server:port/path/package-0.2.whl
(already installed message)
pip install http://server:port/different/path/package-0.2.whl
(already installed message)

[benoit-pierre]

AFAIK, the requirement is prepared, but not installed, because the version is already present.

[benoit-pierre]

How about something along those lines:

 src/pip/_internal/req/constructors.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git i/src/pip/_internal/req/constructors.py w/src/pip/_internal/req/constructors.py
index 4c4641dc..50292ab8 100644
--- i/src/pip/_internal/req/constructors.py
+++ w/src/pip/_internal/req/constructors.py
@@ -176,7 +176,7 @@ def install_req_from_editable(
 
 def install_req_from_line(
     name, comes_from=None, isolated=False, options=None, wheel_cache=None,
-    constraint=False
+    constraint=False, package_name=None,
 ):
     """Creates an InstallRequirement from a name, which might be a
     requirement, directory containing 'setup.py', filename, or URL.
@@ -263,6 +263,8 @@ def install_req_from_line(
                 "Invalid requirement: '%s'\n%s" % (req, add_msg)
             )
 
+    assert package_name is None or req.name == package_name
+
     return InstallRequirement(
         req, comes_from, link=link, markers=markers,
         isolated=isolated,
@@ -292,6 +294,11 @@ def install_req_from_req(
             "which are not also hosted on PyPI.\n"
             "%s depends on %s " % (comes_from.name, req)
         )
+    if req.url:
+        return install_req_from_line(req.url, comes_from=comes_from,
+                                     isolated=isolated,
+                                     wheel_cache=wheel_cache,
+                                     package_name=req.name)
 
     return InstallRequirement(
         req, comes_from, isolated=isolated, wheel_cache=wheel_cache

So basically package @ URL is treated like URL.

[cam72cam]

That works perfectly for my test cases. I'll close my PR if you can open one from that patch.

[pfmoore]

OK, cool. Thanks for clarifying.

I prefer the alternative proposal of determining the version from the filename in the URL, if possible, when parsing a direct URL, just like we do for a dependency picked up of PyPI (getting the version from a .whl filename is well-specified, and we have standard heuristics for doing so from sdists). That sounds like a far better solution than just always installing direct URLs,

[benoit-pierre]

It still needs some work (particularly with respect to extras handling), but I'll see if I can make a proper PR.

[cam72cam]

@benoit-pierre do you think it would be possible to add this to the 18.1 milestone?