Weekly RTs

[gspetro-NOAA]

This discussion is intended to continue the conversation from #2822 .
It sounds like historically, there was no baseline comparison for weekly RTs. However, @DeniseWorthen thought we should add LIST_FILES (and potentially maintain a baseline?).
@jkbk2004 has suggested that EPIC implement the weekly tests via Jenkins so that they can be launched automatically once a week.
I'm trying to gather suggestions and see if we can come to some kind of consensus. If an issue is needed for follow-on work, we can open it once we've settled on a path(s) forward.

Tagging @BrianCurtis-NOAA @dpsarmie @NickSzapiro-NOAA @FernandoAndrade-NOAA @jkbk2004 for additional input.

[dpsarmie]

Thanks for starting this discussion. From my end, we don't need to maintain baselines because they'll probably be changing every week.

In my opinion, the idea of the weekly tests is to make sure that memory and runtime of the runs haven't increased over longer runs (ex. memory leaks), check that the debug and other configurations still running, and check for restart reproducibility.

[NickSzapiro-NOAA]

Don't we need test baselines to check for (restart) reproducibility @dpsarmie ? Could always make locally like rt.sh -c && rt.sh -m

[DeniseWorthen]

@dpsarmie Agreed. I thought the idea would be to run and create a (non-saved) baseline and then re-run with -m.

We'll need a LIST_FILES to compare though, otherwise all we'll know is that the tests ran.

[dpsarmie]

Yep agree with Nick and Denise. In my mind maintaining baselines meant keeping and saving baselines week-to-week.
They should still be generated every week for comparing but they can be deleted once the comparisons are done.
I can go ahead and modify the test files to start adding files for comparison.

[gspetro-NOAA]

@dpsarmie Thanks! Could you also create an issue for that work? And then @FernandoAndrade-NOAA maybe you can create an issue for the Jenkins work?

I think once Dan's PR is in, someone on EPIC can just run the weekly tests even if the Jenkins work isn't done, but obviously the sooner it finishes, the less manual work there is for us.