SSP/SSPMATRIX test problems with new case github workflow

[ekluzek]

This is in ctsm5.3.015

The SSPMATRIX test on Izumi is failing with this problem. I think this is a problem with the new workflow of saving the case details to a local git repository, and likely will be fixed with the next updates to cime.

2024-12-18 13:10:33: ERROR: Command: 'git -C /scratch/cluster/erik/tests_ctsm5315acl/SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int commit -m "
ERROR: RUN FAIL: Command '  /scratch/cluster/erik/tests_ctsm5315acl/SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int/bld/cesm.exe   >> cesm.log.$LID 2>&1 ' failed
See log file for details: /scratch/cluster/erik/tests_ctsm5315acl/SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int.step0-AD/run/cesm.log.707138.izumi.cgd.ucar.edu.241218-131031"' failed with error 'On branch SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int
nothing to commit, working tree clean' from dir '/scratch/cluster/erik/tests_ctsm5315acl/SSPMATRIXCN_Ly5_Mmpi-serial.1x1_numaIA.I2000Clm50BgcCropQianRs.izumi_intel.clm-ciso_monthly.GC.ctsm5315acl_int.step0-AD'

Originally posted by @ekluzek in #2900 (comment)

[ekluzek]

I'm still seeing this in ctsm5.3.016 and think it will take some more work to get working. The SSP and SSPMATRIX tests were based on older testing code and could be refactored to use system_tests_compare_two or system_tests_compare_n.py. The update would probably increase robustness and future protect these tests as well.

The PVT.py test and some others may need similar treatment.

[ekluzek]

One problem I did see is that the final test run is one of the clones made rather than the original test. I think rearranging the order so that the final test run is the original test would help this. That was something arbitrarily done when these tests were made that now with changes has caused a problem. I also realized that the ERI test still does things similar to the SSP tests and modeling what it does might help here. So it might not need a refactoring to use the system_tests_compare*.py infrastructure, and likely that might be easier.

[ekluzek]

I played around with this a bit, but didn't get it working.

I think some code similar to this needs to be added to handle the drv rpointer file.

+        drvrest = "rpointer.cpl"
+        drvrest += self._rest_time
+        self._set_drv_restart_pointer(drvrest)

One catch is that _rest_time doesn't work in above, without using self._set_restart_interval. So that needs to be added in.

[ekluzek]

Running ctsm_sci testlist on ctsm5.3.016 the only unexpected fail was this one:

SSP_Ld4.f09_g17.I1850Clm50BgcCrop.derecho_intel.clm-ciso_rtmColdSSP