Support URLs with arbitrary schemes via plugins

[dborowitz]

What's the problem this feature will solve?

This is essentially the same FR as #10790, with similar motivation: allow users to reference packages and requirements files hosted on cloud storage like Google Cloud Storage and S3.

I have a proof-of-concept implementation involving plugins. I've read enough previous discussions (#10790, #4475, #12766, etc.) to know that this is unlikely to be approved, but this specific approach requires a very minimal stable-API commitment from pip itself, so I thought it worth filing the issue.

Describe the solution you'd like

My prototype is here. In short, it allows users to specify --scheme=<scheme>=<module>:<BaseAdapter subclass> to make <scheme>:// be handled by a specific requests BaseAdapter implementation, dynamically imported at runtime.

What makes this different from most plugin proposals is that most of the stable interface definition is in the requests project, not pip itself; and BaseAdapter itself has barely changed in a decade. The only commitment from pip would be that it continue to use requests, which is fairly minimal--though admittedly maybe still too much.

Alternative Solutions

1. Hard-code specific protocol adapters as discussed in Introduce support for URLs pointing to python packages hosted in cloud provider storage buckets #10790
2. Add protocol support in a wrapper named something like gcs-pip, via monkeypatching PipSession, PoC here
3. Add protocol support in a wrapper, via dynamically downloading URLs and rewriting command lines/requirements files to point to local copies, PoC here (that's for %pip in Jupyter, but a standalone script would be the same).
4. Support custom auth headers as in Allow pip to send arbitrary headers to various hosts #8042 or pip should support custom authentication handlers for private pypi #4475, which would allow using public https:// URLs for cloud providers.

If there's not a clear path forward with the existing pip issues, my team is leaning towards #2.

Additional context

Apologies for the fresh issue; I'd have left a comment on #10790, but it's locked.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[notatallshaw]

What makes this different from most plugin proposals is that most of the stable interface definition is in the requests project, not pip itself; and BaseAdapter itself has barely changed in a decade. The only commitment from pip would be that it continue to use requests, which is fairly minimal--though admittedly maybe still too much.

I have no opinion on plugins, for or against, but I will say that I plan to do a review of how pip uses requests once we are able to upgrade to urllib3 2.x, and see if there is a path forward to removing requests as a dependency and relying on urllib3 directly. This may go nowhere as both the explicit and implicit dependencies may be too great.

But for that, and other reasons, I would be uncomfortable basing a programmatic interface to pip on a vendored library. If plugins are ever agreed upon, I think it's something that must be maintained within pip itself.