`experimental_test_shell_command` should support capture of outputs (test reports, coverage, etc.)

[tdyas]

experimental_test_shell_command currently does not support any capture of outputs (even though the underlying shell backend rules already support that capture). Output capture would permit users to capture coverage output and test reports generated from the test executed via experimental_test_shell_command.

The main UX concern is how to export the captured results. The existing test goal already permits export, so a first solution should probably take advantage of that support.

The test goal internally classifies captured data into several categories. (See the internal TestResult dataclass for particulars.) A first solution could map captured files to one of these categories (or else work would be needed to rework the test goal to better accommodate arbitrary data):

• Coverage
• "XML reports" - Alludes to JUnit-style XML output but really should be renamed.
• Extra output

A naive solution might just expose the shell_command-related output capture fields (e.g., output_files, output_directories) without a mechanism for classifying the results. That would not integrate well with the exiting test goal.

A better solution might have category-specific fields like test_report_outputs, coverage_outputs, and extra_outputs. (This is just a first proposal, we would need to consider how to interact with the output_files and output_directories distinctions.)

[tdyas]

This issue assumes that reuse of outputs from an experimental_test_shell_command execution in other shell_command or adhoc_tool targets is not needed. Instead, the assumption is that test outputs should be captured and exported in the same manner as other tests run via the test goal.

[jcrobak]

I'm not familiar with how the other test outputs work, but it looks like they're mostly controlled by booleans in both golang and python? For the experimental_test_shell_command it would be helpful to be able to capture arbitrary output. Would the files end up in the dist folder or somewhere else?

[tdyas]

I'm not familiar with how the other test outputs work, but it looks like they're mostly controlled by booleans in both golang and python?

Correct. This occurs because those languages are not as general as experimental_test_shell_command and Pants controls the filenames. For example, when Go coverage is enabled, the Pants Go backend's rules control the exact name of the output file to capture.

Would the files end up in the dist folder or somewhere else?

Yes, the test goal outputs under dist/ generally (since it respects the --pants-distdir option).

[jcrobak]

Got it makes sense. The only thing I would clarify is that it could be helpful to capture multiple files/directories. e.g. in the case of cypress tests, screenshots are generated for every test that fails.

[tdyas]

Got it makes sense. The only thing I would clarify is that it could be helpful to capture multiple files/directories. e.g. in the case of cypress tests, screenshots are generated for every test that fails.

The test goal uses Snapshot internally which represents multiple files. So that use case should be straightforward to implement.

[tdyas]

I may just implement support for the "extra outputs" category since test reports and coverage data seem to be specific to specific language backends. If an experimental_test_shell_command test did produce coverage data, it could just be captured as "extra output" and the test goal will output it under dist/ (based on target name usually).

[tdyas]

@jcrobak: Can you test drive #22002 locally and let me know how well (or not) it works for you?