Add support for URL dependencies (at least for wheels)

[charliermarsh]

• VCS:
  • Start with just git+
  • The URL must point to a source distribution
  • If the URL ends with a hash, we can use that as the cache key
  • If the URL ends with anything else, we need to resolve it to a commit, then use that as the cache key (and we need to do this lookup every time)
• URL dependencies can be a source distribution or a wheel
  • For URL dependencies, we should cache based on the HTTP semantics
  • Empirically, it seems that the URL must end in .whl to be considered a wheel; otherwise, it’s considered a source distribution
  • We could even start by not caching anything here

[charliermarsh]

I need to look at how some other tools handle this (e.g., Bun, pip, Poetry).

[charliermarsh]

oven-sh/bun#1875 (comment)

[charliermarsh]

pypa/pip#10075

[charliermarsh]

pypa/pip#11164

[charliermarsh]

Lots of good discussion in here: pypa/pip#10564 (comment)

[charliermarsh]

Gonna propose something here...

[konstin]

Unlike pip, we should remember the url contents we installed (e.g. through an etag or another http caching mechanism), check the url every time and reinstall if they changed. For locked requirements.txt, we would need hashes.

[charliermarsh]

Yeah we need a clear mechanism for this. But the pip issues also have a lot of discussion around what happens when you change from a URL dependency to a version dependency, etc.

[charliermarsh]

Probably the biggest remaining "feature" (with the rest of the milestone being largely focused on testing, performance, and polish).

[charliermarsh]

Another challenge here is that we need to fetch and build the distribution in order to know the version.