CIME needs updating for reliable ERR and IRT tests #3374
Description
Originally posted by @samsrabin in #3354 (comment)
@jedwards4b Is it possible there's a race condition in the IRT SystemTest? I had one passing at 8a9c25d but then failing when I tried later. This prompted me to try five more replicates, of which only one passed. Here are the directories and their results:
/glade/derecho/scratch/samrabin/tests_0728-125516de/IRT_Ld11.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-default.0728-125516de_int: PASS/glade/derecho/scratch/samrabin/tests_0729-100052de/IRT_Ld11.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-default.0729-100052de_int: FAIL/glade/derecho/scratch/samrabin/tests_0729-102142de/IRT_Ld11.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-default.0729-102142de: FAIL/glade/derecho/scratch/samrabin/tests_0729-102202de/IRT_Ld11.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-default.0729-102202de: FAIL/glade/derecho/scratch/samrabin/tests_0729-102220de/IRT_Ld11.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-default.0729-102220de: FAIL/glade/derecho/scratch/samrabin/tests_0729-102240de/IRT_Ld11.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-default.0729-102240de: FAIL/glade/derecho/scratch/samrabin/tests_0729-102301de/IRT_Ld11.f10_f10_mg37.IHistClm60BgcCrop.derecho_intel.clm-default.0729-102301de: PASSThe failures all have this in
run/case2run/drv.log*:read rpointer file = rpointer.cpl.1850-01-05-00000 (esm_time_mod.F90:esm_time_clockInit) ERROR rpointer file rpointer.cpl.1850-01- 05-00000 not found
Originally posted by @jedwards4b in #3354 (comment)
It's not a race condition exactly, it's a faulty method of sorting the restart directories by using mtime. I am experimenting with an alternate method of sorting and will let you know if it's more consistent. By the way I noticed that the ERR test was doing this too, but wrote it off to human error until you said something - thanks.