Feature request: Collect files generated during tests (e.g. coverage data)

[timrid]

Problem

cibuildwheel runs the test suite from a temporary working directory, and on some platforms that directory lives in an environment the host cannot reach:

• Android: tests run inside an emulator. The host has no access to the emulator's filesystem, so any file the tests produce is lost.
• iOS: tests run inside a simulator. The host can technically reach the simulator's filesystem, but locating and extracting files manually is awkward.
• Linux: tests run inside a Docker/Podman container, so generated files are discarded when the container is torn down.

The practical consequence: on iOS and Android there is currently no way to get coverage information out of a test run at all. Even on Linux it requires platform-specific workarounds.

There is no unified, portable way for a package author to say "after the tests run, give me these files back".

Proposed solution

Add a new option that lists files produced by the test run and has cibuildwheel copy them out of the (sandboxed) test working directory into a directory on the host. Because the test working directory mirrors the project layout (it is populated via test-sources), authors can refer to outputs using the same relative paths they would use when running the tests locally — including on mobile, where {project}/{package} placeholders are not available.

New configuration / CLI options

• test-outputs (list of paths /globs, like test-sources)
  Files to collect from the test working directory after the test command runs. Glob patterns are supported (e.g. .coverage*, **/junit.xml) (Edit: I think globs are hard to implement on e.g. Android or using docker cp, so we should not allow them). If a declared pattern produces no matches, cibuildwheel emits a warning and continues (the artifact is treated as optional).
  Env var: CIBW_TEST_OUTPUTS

• test-output-dir (string, default ./test-outputs)
  Where the collected files are written. To avoid collisions, outputs are namespaced per build identifier (e.g. cp312-manylinux_x86_64), since a single cibuildwheel invocation runs the tests many times across Python versions and architectures. A {identifier} placeholder is supported so the layout is configurable; the effective destination is: {test-output-dir}/{identifier}/<relative path of the output file>
  Env var: CIBW_TEST_OUTPUT_DIR.

Per-platform implementation

The mechanism differs only in how files leave the sandbox; the option and the resulting on-host layout are identical everywhere.

Native (macOS, Windows) and Pyodide:
The test working directory is on the local filesystem. After the test command finishes (and before the temp dir is cleaned up), copy the matching files into the per-identifier output directory.

Linux (Docker / Podman):
Resolve the patterns inside the container's test working directory and copy each match out to the host (e.g. via docker cp / podman cp) before the container's temp directory is removed.

Android:
I created python/cpython#149271 that allows to copy files back from the Android emulator. Each test outputs entry would be forwarded to the testbed runner as an output argument; cibuildwheel then places the returned files in the per-identifier output directory.

iOS:
The simulator testbed would gain an equivalent output argument (kept in parity with the Android testbed flag), and cibuildwheel wires it up the same way.

Related issues

• gh-149113: Add option to copy back data from Android testbed emulator to build machine python/cpython#149271 (review)

These issues are requesting similar features but are docker specific. iOS and Android are not mentioned there:

• RFE: env var for post build and test command #115
• Way to publish other files/binaries as GH Release from each build? #363
• Is there a way to extract non-wheel outputs from a cibuildwheel Docker container? #1030
• Allow copying artifacts to /output in case of test failure on docker #1756

[joerick]

Thanks for writing this up. I think there's space for a feature here.

I think your design above makes a lot of sense. I note that this design doesn't cover the feature request in #363 or #1030, as these issues are talking about bringing back build artefacts, not test phase ones. Those ones are often worked around using the /host mount.

So this feature is really much more specific to test coverage/reports. But that's ok, because it allows us to do the emulator/simulator extraction that you mention.

So my only question is- do we need the test-output-dir option at all? What if we just put the extracted files into the wheel output dir (./wheelhouse by default), within identifier subfolders? Is there a use case that would be made possible or lots simpler with this option?

[agriyakhetarpal]

I am not the author here, but I thought I could speak to the need for a test-output-dir option: it would be useful because a folder named wheelhouse suggests it only stores wheel files. I haven't attempted to run any numbers or evaluate my hunch with any concrete code search results as such, but I think adding non-wheel files could break release workflows when they're present in the folder to be uploaded to PyPI/the package repository. Most people should be copying/moving wheelhouse/*.whl between artifacts, but I imagine we will find some cases of a more rudimentary wheelhouse/* too. In that sense, it's probably best to provide safer defaults at the expense of adding an extra option.