Too many Test cases?

[Lewih]

Is it possible to collapse many test cases generated by a parametrised test into a single more sophisticated test while preserving a meaningful reframe log output?

Is it necessary? Different test groups can be run in different moments (using tags) reducing the number of test cases run simultaneously.

[stevenvdb]

To be clear: I have no problem with many test cases and I completely understand the logic that you want a test to check exactly one thing. What we should keep an eye on, is that the number of batch jobs submitted when running the test suite does not grow exponentially. There is for example a separate batch job for each environment variable that is tested. Currently there are 40 batch jobs submitted when running the entire suite, which is still manageable. It would be nice if ReFrame would support running multiple tests within a single job for this purpose.

[Lewih]

Surely would be handy to have different test cases in a single job. investigating.

[smoors]

an alternative would be to launch ReFrame from inside a job, and use the 'local' partition to run the tests.
of course, that does not work for tests that must run in the login node, but those test run in the 'local' partition anyway.

[Lewih]

Let's use @stevenvdb 's idea: performance test report as a mean to check True and False validation tests.

From Slack: No-one is aware of a way to submit multiple tests in a single job.

[Lewih]

they are working on a more complete test logging.

from slack:

Technically, full test logging is currently not available except through performance logging. We have an feature request to change this and support proper test case logging, regardless of whether it’s a performance test or not.

link

[stevenvdb]

For now we will keep one test per job. Perhaps in the future this will change if ReFrame offers better support for it. This can be closed.