`--keyring-provider` does not work with `build-system.requires`

[AngellusMortis]

Description

We are using Google Artifact Registry to install packages. Using a combination of --no-input and --keyring-provider=import works for installing wheels from GAR, but it does not work when installing a package from source as the requirements in the build-system.requires does not use the keyring auth.

Expected behavior

When pip install requirements for build-system.requires, it should use the same keyring behavior as installing a package normally.

pip version

23.3.1

Python version

3.10.13 (Docker, official image)

OS

Debian Bullseye (Docker, official image)

How to Reproduce

1. Add a private repo that requires keyring auth to pip config:

[global]
index-url = https://url/to/repo

2. Install keyring and setup keyring auth (example with Google Artifact Registry)

pip install keyring keyrings-google-artifactregistry-auth

3. Verify keyring auth is working with --no-input (install package with wheel)

pip install --no-input --keyring-provider=import requests

4. Try to install a package from source/tarball. flake8-mutable looks like it works as a test package as it does not have a wheel, but you can pip install a source directory as well.

pip install --no-input --keyring-provider=import flake8-mutable

Output

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [35 lines of output]
      /usr/local/lib/python3.10/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
        warnings.warn(
      ERROR: Could not find a version that satisfies the requirement pytest-runner (from versions: none)
      ERROR: No matching distribution found for pytest-runner
      Traceback (most recent call last):
        File "/usr/local/lib/python3.10/site-packages/setuptools/installer.py", line 82, in fetch_build_egg
          subprocess.check_call(cmd)
        File "/usr/local/lib/python3.10/subprocess.py", line 369, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/usr/local/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp01v07qq3', '--quiet', 'pytest-runner']' returned non-zero exit status 1.

      The above exception was the direct cause of the following exception:

      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/tmp/pip-install-35h5zefc/flake8-mutable_09876f5ce1344b51b1270e2d9407a954/setup.py", line 5, in <module>
          setuptools.setup(
        File "/usr/local/lib/python3.10/site-packages/setuptools/__init__.py", line 86, in setup
          _install_setup_requires(attrs)
        File "/usr/local/lib/python3.10/site-packages/setuptools/__init__.py", line 80, in _install_setup_requires
          dist.fetch_build_eggs(dist.setup_requires)
        File "/usr/local/lib/python3.10/site-packages/setuptools/dist.py", line 875, in fetch_build_eggs
          resolved_dists = pkg_resources.working_set.resolve(
        File "/usr/local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 789, in resolve
          dist = best[req.key] = env.best_match(
        File "/usr/local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1075, in best_match
          return self.obtain(req, installer)
        File "/usr/local/lib/python3.10/site-packages/pkg_resources/__init__.py", line 1087, in obtain
          return installer(requirement)
        File "/usr/local/lib/python3.10/site-packages/setuptools/dist.py", line 945, in fetch_build_egg
          return fetch_build_egg(self, req)
        File "/usr/local/lib/python3.10/site-packages/setuptools/installer.py", line 84, in fetch_build_egg
          raise DistutilsError(str(e)) from e
      distutils.errors.DistutilsError: Command '['/usr/local/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmp01v07qq3', '--quiet', 'pytest-runner']' returned non-zero exit status 1.
      [end of output]

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[Darsstar]

As the author of the --keyring-provide flag I recommand you use the subprocess provider and run pip config set --global global.keyring-provider subprocess. (you can use --user instead of --global if you prefer.) You'd need to install keyring somewhere on the PATH, for which I recommend Pipx...
https://pip.pypa.io/en/stable/topics/authentication/#using-keyring-as-a-command-line-application

I guess you could try to disable the isolated build feature, but it is not something I recommend doing.

[AngellusMortis]

That does hot help solve the listed issue in any way. Pip would still not use the keyring for installing the build system requirements.

[Darsstar]

Oh, right. The Python version is probably not new enough that the ensurepip module does not ship with Pip 23.1+ yet. (The result is new venvs by the venv module come with an old Pip version.)

Edit: hmm nope 3.10.13 should be new enough...

You could patch it yourself, I did that for myself and several colleagues, if moving to a more recent Python version isn't possible.

If you figure out something better please let me know. This is the best situation I managed to create after multiple PRs.

Happy new year!

[mikhail-mmlc]

The workaround is to set the keyring provider in the pip config file instead of using the --keyring-provider option.
For example, by adding the following:

[global]
keyring-provider = import

to $HOME/.config/pip/pip.conf, the import provider is enforced for both direct and build requirements.

[Darsstar]

The workaround is to set the keyring provider in the pip config file instead of using the --keyring-provider option. For example, by adding the following:

[global]
keyring-provider = import

to $HOME/.config/pip/pip.conf, the import provider is enforced for both direct and build requirements.

pip config set --global global.keyring-provider import would be how to achieve that via the CLI.

And that is actually how I intended most people to configure their system. (Well, with subprocess as the value.)

[mikhail-mmlc]

It's great that multiple methods are available to set the keyring provider value.
However, there appears to be an issue as values from the configurations and environment variables are being passed (indirectly but still) to the BuildEnvironment, along with values from options such as --index-url and --verbose, but not --keyring-provider value, which is probably an oversight.
I think the issue is larger than just the --keyring-provider since the BuildEnvironment doesn't have direct access to command-line options. This leads to inconsistencies in the options compared to what the main pip command calls and the options it passes to the subprocess that installs build requirements.

[pfmoore]

I think the issue is larger than just the --keyring-provider since the BuildEnvironment doesn't have direct access to command-line options.

Correct. But equally, not all command line options should be passed to the build environment. I think there's already an issue regarding the broader problem here, but I don't have time right now to search for it, sorry.