Docker-based integration tests

[chrahunt]

What's the problem this feature will solve?

When writing automated tests for issues that depend on system setup or environment, stubbing out or mocking the related interfaces can be difficult or less robust than full integration tests:

1. changing the configuration, for example to use a specific configuration file, changes the actual code paths taken
2. mocking system-related functions (e.g. os.* or ssl.*) creates a maintenance overhead if we change the implementation later
3. the isolation may be incomplete (see Isolation issues in tests/functional/test_help.py #5957)

Describe the solution you'd like

Use Docker to execute integration tests that can be invoked locally (opt-in) and in CI.

This should not impact existing functional and unit tests, but enable new test strategies for scenarios that require more isolation or prerequisites that depend on global state in the OS, filesystem, or network.

Examples that could benefit:

1. Installing wheel with different users  #2125
2. Isolation issues in tests/functional/test_help.py #5957
3. virtualenv with --system-site-packages breaks pip's build isolation #6264
4. noexec /tmp causes pip install to miss chmod +x on scripts included in wheel file #6364
5. Test failures occur when running from inside a linked worktree #6707

The specific tasks required to close this issue would probably be:

1. Basic interface for interacting with and executing commands in a running container - similar functionality to tests.lib.PipTestEnvironment.
2. At least one test showing usage of the above, e.g. demonstrating pip behavior which requires reading edited global pip.conf
3. Integration with CI

Alternative Solutions

1. Continue using strictly in-process/environment-based isolation and redirection for tests.
2. Using Docker-based integration tests, but running the tests from inside the container. This may be slower and depending on the specific test case we may need to opt-out of currently global behavior (e.g. any fixture with autouse=True). This would also need some higher-level orchestration or commands to actually execute and accumulate the test output.
3. Keeping integration tests in a separate repository.

Additional context

Azure pipelines support Docker for Linux (via ubuntu-16.04) and Windows (via win1803) per this page.

There is not a clear winner with regards to pytest plugins, but having a simple helper that executes commands in a container spawned from a pre-defined image (or one of a set created for the test) should not be too much.

[pfmoore]

How would this impact people running the tests locally? Would we then require docker? (A non-trivial requirement on Windows, at least)

[chrahunt]

If this is opt-in as proposed then it should have no impact unless explicitly requested (e.g. tox -e docker).

In terms of requirements on contributors in general, I think it's OK to let CI handle testing that is not possible locally, though the feedback loop will be longer than if running locally. We can address this in developer documentation or a dedicated issue label for issues related to pain points in development environment setup.