Regression tests

[JessicaMeixner-NOAA]

A place to start the discussion on regression test improvements. This could include:

• How to make them run more efficiently
• To cover more code
• Adding tests
• etc

[aliabdolali]

These are my ideal regtest:

• We need regtest to check the compilation of all possible switch combinations. It just needs a dictionary and parallel build of cmake or what @aronroland and @MathieuDutSik did a long time ago but with cmake.
• instead of ideal cases, we need to have cases with observations. ERDC/Aron/myself did it for vegetation, setup, triad, and breaking, .... We found it very useful, and having more of them (ONR cases) will guarantee the accuracy of the model.
• We need a few large cases to check the scalability and performance of UKMET GPU work @ukmo-ccbunney .
• We need some tests to check operational requirements (coherency, ...)
• We need a clean-up of regtest system.
• We need tools to compare relative differences, our matrix.comp is not adequate.

[aliabdolali]

Comment from @MathieuDutSik

With regard to regtest for compilation, I do not think that all the cases can be tested. There is just a combinatorial explosion of possibilities since WW3 has too many switches.

What I did instead is that I wrote a python script that selects switches at random according to the constraint that are mandated and then do the compilation. You can run that for a night and then you will find stuff. I found several bugs for which I submitted the solution last year.

There are possible ways to improve the script for example by detecting unused variables. Also, there are some memory inconsistencies between arrays that can potentially cause crashes.

In any case, I believe Aron Roland mentioned many years ago the need to use compiler sanitizers for detecting bugs in WW3. The recommendation is still relevant in my opinion and running the small-scale examples with say less than 1000 nodes with it would surely uncover bugs.

[aronroland]

Hi @MathieuDutSik, this will be done and we will make the relevant issues.