Description
Description of Problem:
If a file has multiple tests and one of them is skipped, all subsequent test in the file is skipped.
OpenSCAP Version:
Building from main with 74043f7.
Operating System & Version:
Debian Bookworm
Steps to Reproduce:
After a normal build, run ctest -R test_ds_misc --verbose
Log output
Test project /openscap/build
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 176
Start 176: DS/test_ds_misc.sh
176: Test command: /openscap/tests/DS/test_ds_misc.sh
176: Working Directory: /openscap/build/tests/DS
176: Environment variables:
176: srcdir=/openscap/tests/DS
176: top_srcdir=/openscap
176: builddir=/openscap/build
176: Test timeout computed to be: 10000000
176: + sds_external_xccdf_in_ds
176: TEST: sds_external_xccdf_in_ds
176: --- Starting Evaluation ---
176:
176: Title
176: Rulxccdf_moc.elpmaxe.www_rule_1
176: Respass
176:
176: OVAL Results are exported correctly.
176: XCCDF Results are exported correctly.
176: Result data stream exported correctly.
176: RESULT: PASSED
176:
176: + sds_external_xccdf
176: TEST: sds_external_xccdf
176: --- Starting Evaluation ---
176:
176: Title
176: Rulxccdf_moc.elpmaxe.www_rule_1
176: Respass
176:
176: OVAL Results are exported correctly.
176: XCCDF Results are exported correctly.
176: Result data stream exported correctly.
176: RESULT: PASSED
176:
176: + sds_tailoring
176: TEST: sds_tailoring
176: Document type: Source Data Stream
176: Imported: 2024-05-24T06:18:36
176:
176: Stream: scap_com.example_datastream_with_checklist
176: Generated: 2016-09-26T15:00:00
176: Version: 1.2
176: Checklists:
176: Ref-Id: scap_com.example_cref_xccdf_01
176: Status: incomplete
176: Resolved: false
176: Profiles:
176: Referenced check files:
176: oval
176: system: http://oval.mitre.org/XMLSchema/oval-definitions-5
176: Checks:
176: Ref-Id: scap_com.example_cref_oval_01
176: No dictionaries.
176:
176: Stream: scap_com.example_datastream_with_tailoring
176: Generated: 2016-05-09T15:00:00
176: Version: 1.2
176: Checklists:
176: Ref-Id: scap_com.example_cref_tailoring_01
176: Benchmark Hint: #scap_com.example_cref_xccdf_01
176: Profiles:
176: Title: Tailoring profile
176: Id: xccdf_com.example_profile_tailoring
176: Ref-Id: scap_com.example_cref_xccdf_01-source
176: Status: incomplete
176: Resolved: false
176: Profiles:
176: Referenced check files:
176: oval
176: system: http://oval.mitre.org/XMLSchema/oval-definitions-5
176: Checks:
176: Ref-Id: scap_com.example_cref_oval_01-source
176: No dictionaries.
176: --- Starting Evaluation ---
176:
176: TitRule 2
176: Rulxccdf_com.example_rule_2
176: Respass
176:
176: XCCDF Results are exported correctly.
176: RESULT: PASSED
176:
176: + eval_simple
176: TEST: eval_simple
1/1 Test #176: DS/test_ds_misc.sh ...............***Skipped 1.45 sec
The following tests passed:
DS/test_ds_misc.sh
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 1.46 sec
The following tests did not run:
176 - DS/test_ds_misc.sh (Skipped)
From the output, you can see that the last test executed is
openscap/tests/DS/test_ds_misc.sh
Line 293 in 74043f7
This is the fourth, out of 23 test in tests/DS/test_ds_misc.sh. This behavior can also be seen from GitHub Actions if we look at how long time it takes to test this file. Fedora runs all of the tests and uses 7.93 sec, while Ubuntu, which skips a test, only uses 1.29 sec. The timing itself is not enough, but combined with the attached log, I believe it is obvious that all subsequent tests is also skipped during CI pipeline.
Actual Results:
The test-suite doesn't run all the tests, if one or more test is being skipped.
Expected Results:
All tests except the ones to be skipped should run, regardless if a previous test has been skipped.