proposal: replace keyring dependency with generic pip-defined credential helper API

[rittneje]

Currently pip directly imports keyring in order to use it for authentication. Furthermore, to avoid a performance hit in the majority case, it will only consult keyring if the original request fails with 401. This has caused a few issues:

1. Some users do not want pip to use keyring, even if it is present on the system. Keyring support should require an --enable-keyring flag #8719
2. When interacting with a private PyPI repo, the initial unauthenticated request is undesirable. Add flag to force pip to always fetch password from keyring #10269
3. Due to a missing keyring feature (at least on macos), the username still has to be specified in the index url. This in turn causes pip's credential cache to not function properly. fix auth cache to allow for username in index url #10288
4. Relying on keyring leads to a chicken-or-the-egg bootstrapping issue. You need pip in order to install keyring, but if you need to auth with a private PyPI repo to do so, now what?
5. In general, having pip directly rely on a specific external module seems undesirable. What happens if keyring falls out of vogue, or is abandoned?

I would like to propose that the direct dependency on keyring be removed, and replaced with a more generic concept, akin to docker's credential helper concept. https://docs.docker.com/engine/reference/commandline/login/#credentials-store

The basic idea is that the pip configuration be able to specify a credential helper library for a given PyPI hostname. If there is no entry for a hostname, then requests are sent without any authentication. This allows the majority case of talking to public PyPI to continue to work as it does today with no performance hit. If a hostname does have an entry, then it is expected to map to a library that exposes a get_pip_credentials(hostname) function, which takes the download hostname and returns the username and password to use. The user just needs to arrange for this library to be importable, so for example they could write a custom library and add it to their PYTHONPATH.

For keyring (excluding macos), a very simple adapter around the keyring API could be created, and ideally could even be part of keyring itself. For macos, I can make my own shim to pull the keyring username from some environment variable of my choosing. For CI/CD, a simple credential helper could be made to pull the username and password from pre-canned environment variables. (Since this logic lives entirely outside pip, the issue of multiple index urls mentioned here is not relevant.)

If a user doesn't want to use keyring, they simply would not specify it in their pip configuration. That means that by default, keyring would never be used at all. But it also means that users that do want to use it are not required to pass some flag to pip all the time.

If the credential helper API is standardized, then it can be shared with other dependency management tools such as pipenv, instead of the situation today where the two have completely different approaches. (For reference, most docker-like tools, such as skopeo and buildkit, will abide by your credential helpers, thus providing a consistent experience.)

[uranusjr]

Sounds like a good thing to me. Pip historically tend to expose too much dependencies' internals to the user, and they generally become a problem down the road (the most significant is at the moment our dependence on pkg_resources and distutils). However, keyring being not an essential part of the packaging mechanism, it is more difficult to gather maintainer interest for this (keyring support was contributed by its author). Would you be interested in working on this? I would be happy to help review the design and implementation.

[pfmoore]

One constraint I would note is that we've traditionally been very explicit in not supporting people writing code that uses pip's internal API. From a brief scan of the proposal here, it seems to do a good job of limiting the interface, and I much prefer an approach like this than either the current "magical behaviour change" if keyring is installed, or the other proposals for command line options. So I'm basically in favour of this, as long as care is taken to make sure it doesn't set a precedent that people can write code against pip's internals.

[rittneje]

@uranusjr @pfmoore I've raised an initial draft PR with just the changes for auth.py. I haven't wired it through to actually pull the mapping from pip.conf yet. Been trying to piece together how that actually works.

On that note, would something like this be reasonable? I'm not sure of the best way to do it given the constraints of the pip.conf format.

[global]
credential-helper = my-private-repo.com=keyring
credential-helper = another-private-repo.com=some_custom_module

[sbidoul]

Would it make sense to use Entry Points as discovery mechanism for credential helpers ? This implies delegating the configuration mechanism to the helpers themselves.

[pradyunsg]

I think keeping the configuration of the credentials helper in pip is cleaner, since it allows arbitrary executables to be the helpers (i.e. allowing non-Python based implementations), doesn't require them to manage duplicated configuration stuff, and lets us the dispatch to different credential managers based on the domain.

[sbidoul]

allowing non-Python based implementations

a small python wrapper could call non-python implementations ?

doesn't require them to manage duplicated configuration stuff

fair

dispatch to different credential managers based on the domain

credential managers could accept the domain as argument, and pip could try them all until one knows about the domain ?

OTOH returning a user and password is only one possible authentication mechanism (people have been asking for ways to inject http headers etc). So the authentication and possibly in the future the http session mechanisms are bound to grow in complexity, which may or may not make them difficult to handle via pip's configuration.

[pfmoore]

... on the other hand, if we use an entry point, someone could write a helper (in Python) that did all of that config management, delegation to arbitrary executables, dispatching based on domain, etc.

That saves us having to get into the business of designing credential management schemes, and lets interested parties come up with their own mechanisms. I'd expect "standard" helpers to get developed and published, so this wouldn't require everyone to write their own.

(Basically, I'm arguing that we have the bare minimum in pip, and let others do the work 😉)

[uranusjr]

The problem (I think) with supporting alternative authentication means (e.g. adding headers) is we’ll basically need to expose the entire session interface, which is another internal thing we don’t really want to rely on. If we want this to be as pluggable as possible, pip would need to provide an abstraction layer over requests.Session, which could be significant work (both initially and in maintenance).

[rittneje]

I don't entirely understand what an entry point is, but it kind of feels like it would be the same situation where it gets used just because it is on the system, and not because the user actually wants it. To me the advantage of having it be explicitly part of the pip config is that the user has full control over what pip does.

I can't speak to the need for arbitrary HTTP headers since it isn't a part of my specific use case, but we could have the return value from the helper be a dict rather than just the username/password tuple directly. That would for example allow it to return a "headers" field in the future too without breaking anything.

[rittneje]

@uranusjr @pfmoore Any further thoughts on this?

[pradyunsg]

credential managers could accept the domain as argument, and pip could try them all until one knows about the domain ?

This sounds very inefficient, and a painful thing for users to debug when it's misbehaving.

(Basically, I'm arguing that we have the bare minimum in pip, and let others do the work 😉)

While I agree with this principle in general, I think we shouldn't do that here.

This has many similarities to PEP 516 vs PEP 517 IMO -- have-everyone-solve-the-same-problem for a simpler core, or a solve-the-problem-in-one-place for simplicity everywhere else. I prefer the latter, and that's also the route we took with PEP 517. :)

The problem we have at hand is exactly the same as what docker has (minus the fact that we need to move away from keyring). The exact problem at hand has already been solved by the thing we're modelling against and, honestly, I don't think we're going to be making major usability or maintainability breakthroughs over what they have.

This change would also give us a clear point to make this explicitly opt-in as well, which is nice. :)

IMO, one of the nice things that we can do here is piggy back on the ecosystem of credential helpers that have been built for that -- instead of coming up with our own protocol here, I think we can basically reuse docker's protocol for this. Allowing downstream users to do something like ln -s /usr/bin/docker-credential-foo /usr/bin/python-credential-foo (yes, this is unix-specific but equivalents exists on Windows and copy-renaming executables will likely work too) is better for everyone involved IMO -- we reduce the amount of busy-work the ecosystem has to do overall and we also make life easier for ourselves by using an existing definitely-working solution instead of needing to come up with our own and solve issues as they prop up (see keyring integration).

[pfmoore]

I think we can basically reuse docker's protocol for this

Is this the same as the git credential manager protocol? Do you have a link to a spec? This is the basis of the git one.

While I don't want to over-engineer too much here, if this is a general issue, should we go the extra step and standardise? After all, we try to make a point of saying that pip uses standards-based solutions so that we avoid the problem of having so much implementation-defined behaviour that nobody will ever stand a chance of writing a pip replacement.

It could be as simple as "Use the docker (git) protocol [add a link to the spec here], using the name pypa-credential-foo" (note the use of "pypa" rather than "pip").

(BTW, if standardising is "too complicated" for something this simple, I'd argue that means our process is bad, not that this doesn't need to be standardised!! In which case let's fix our process...)

[pradyunsg]

Based on a quick reading, it looks like the protocols are different.

I think what we should use for the prefix is python if we’re gonna go through standardisation.

@pfmoore If I write the PEP for this, would you be willing to sponsor it?