For any retest of C++ test suite, the gtest output xml file gets overwritten. This is problematic, for example, if the test suite contains a flaky test. It could fail on the first run, but pass on the second run (given that --retest-until-pass is passed to colcon test). The generated XML output file contains only test results of the last test execution, any previous test exeuction gets overwritten. The only way to detect such cases would be to manually parse the output of colcon test, line by line.
I would suggest to generate separate XML files for each run, e.g.:
build/<pkg>/test_results/<pkg>/<test_suite_name>_run_1.gtest.xml
build/<pkg>/test_results/<pkg>/<test_suite_name>_run_2.gtest.xml
build/<pkg>/test_results/<pkg>/<test_suite_name>_run_3.gtest.xml
...
The problem is that all previous files must be deleted before calling the test suite, as there might be only one test run, while the previous one had multiple ones, and those previous XML files are then being copied to the Testing folder.
Ideally, the output files should be merged, but since gtest does not allow to merge output files, that operation requires manual XML parsing and generation.
For any retest of C++ test suite, the gtest output xml file gets overwritten. This is problematic, for example, if the test suite contains a flaky test. It could fail on the first run, but pass on the second run (given that
--retest-until-passis passed tocolcon test). The generated XML output file contains only test results of the last test execution, any previous test exeuction gets overwritten. The only way to detect such cases would be to manually parse the output ofcolcon test, line by line.I would suggest to generate separate XML files for each run, e.g.:
The problem is that all previous files must be deleted before calling the test suite, as there might be only one test run, while the previous one had multiple ones, and those previous XML files are then being copied to the
Testingfolder.Ideally, the output files should be merged, but since
gtestdoes not allow to merge output files, that operation requires manual XML parsing and generation.