Avoid importing buildenv and metadata modules during autocompletion

[danielhollas]

Speed up tab autocompletion by:

• Moving get_runnable_pip to utils.misc module to avoid importing build_env module. (Per this comment Speedup startup time #4768 (comment))

• Lazy importing metadata module (which transitively imports email and urllib.request which are slow)

• Lazy importing packaging.requirements for another 5% (4ms speedup). I also looked into speeding up packaging.requirements itself, but it would be much more work than the simple change in this PR.

This also speeds up pip --help and possibly other commands.

Benchmarks

❯ hyperfine -w 5 "pip --help"
Benchmark 1: pip --help
  Time (mean ± σ):     110.4 ms ±   7.5 ms    [User: 88.5 ms, System: 20.8 ms]
  Range (min … max):    91.1 ms … 124.3 ms    27 runs
 
❯ git switch main
❯ hyperfine -w 5 "pip --help"
Benchmark 1: pip --help
  Time (mean ± σ):     133.2 ms ±   9.9 ms    [User: 107.0 ms, System: 24.8 ms]
  Range (min … max):   107.7 ms … 144.1 ms    21 runs

TODO:

• Add News entry
• Add benchmark to PR description

Part of #4768

[danielhollas]

It's quite unfortunate to have this import in the middle of the function, but the speedup for autocompletion is quite significant so I think it's worth it. We could also extract the code in this branch into a standalone function, but it seemed like more work than it's worth.

[danielhollas]

This lazy import should be safe as it is only triggered during tab autocompletion.

[danielhollas]

Not a huge fan of this, but InvalidInstalledPackage should hopefully be a rare code path.

[danielhollas]

This is more dicey in terms of security, as InvalidInstalledPackage is used in resolvelib. One solution would be to move the definition of this exception down to the resolvelib module itself?

❯ rg InvalidInstalledPackage
exceptions.py
803:class InvalidInstalledPackage(DiagnosticPipError):

resolution/resolvelib/factory.py
26:    InvalidInstalledPackage,
287:                raise InvalidInstalledPackage(dist=installed_dist, invalid_exc=e)

resolution/resolvelib/candidates.py
16:    InvalidInstalledPackage,
421:            raise InvalidInstalledPackage(dist=self.dist, invalid_exc=exc) from None

[notatallshaw]

I've not reviewed this yet but be aware any lazy imports need to be extremely carefully done as there is a potential security issue:

1. pip installs wheel
2. wheel overrides pip namespace
3. pip calls function that lazy imports
4. Arbitrary code execution happens from just running pip install on a wheel

So any lazy import that happens needs to be verified it can not happen after a wheel is installed and before the command finishes executing.

[danielhollas]

@notatallshaw thanks! Out of the two lazy imports here, one should be safe, the other one I am not so sure. Please see comments inline.