Add test coverage

[chrahunt]

What's the problem this feature will solve?

Currently, pip does not have any coverage tracking integrated into its test suite. This comes with a few downsides:

1. Not possible to quickly identify gaps in test coverage
2. Can't use test techniques like mutation testing, which depend on coverage data
3. Can't use it as a factor for PR evaluation

The current tox environment configured with coverage-py3 is limited to unit tests only.

Describe the solution you'd like

In a separate tox target, configure pytest-cov/coverage to:

1. record coverage details for the pip package and sub-packages/modules, as invoked by:
   1. unit tests directly
   2. tests using the script/virtualenv pytest fixture (see tests/conftest.py -> virtualenv_template)
   3. pip itself, as part of build_env.BuildEnvironment.install_requirements
2. exclude vendored libraries
3. exclude type-only code (guarded with if MYPY_CHECK_RUNNING)
4. exclude Python 2 on Python 3 and vice-versa
5. combine the coverage results for all pips, which may have different paths
6. support getting coverage while using local paralellization with xdist (i.e. -n option)

If it helps, this could be done in Python 3 only (at first).

I have a basic first implementation here, but when combining the coverage reports the records from the integration tests seemed to be dropped.

Alternative Solutions

1. Do nothing, which comes at the cost of not having the initially mentioned capabilities

Additional context

• Add a coverage job for Python 3 #4862 - coverage targeting unit tests added
• twitter

[chrahunt]

I'd call this done once we have it mentioned in the docs, along with the instructions for generating the report as mentioned in #7416.

[pradyunsg]

I think there's a couple more things to do before we close this:

• Document how to generate coverage reports / run tests while tracking coverage.
• Document how we treat coverage during development (in general, PRs that make changes should strive to increase the coverage not decrease it?).