Model does not restart reproducibly in debug mode when initially starting from GFS initial conditions

[spencerkclark]

I encountered this issue in #379 when trying to run the test_restart_reproducibility test with the fortran model compiled in debug mode. For the time being I will simply skip these tests, but we may at some point look into the cause. A clue is that these tests pass if I remove the -finit-logical=true from the FFLAGS_DEBUG variable in the configure file:

diff --git a/FV3/conf/configure.fv3.nix b/FV3/conf/configure.fv3.nix
index d1b36993..18f1d58e 100644
--- a/FV3/conf/configure.fv3.nix
+++ b/FV3/conf/configure.fv3.nix
@@ -68,7 +68,7 @@ endif
 
 FFLAGS_OPT = -O2 
 FFLAGS_REPRO = -O2 -g -fbacktrace
-FFLAGS_DEBUG = -O0 -g -fbacktrace -fno-fast-math -ffree-line-length-none -fno-backslash -pedantic -Waliasing -Wampersand -Wline-truncation -Wsurprising -Wtabs -Wunderflow -fdump-core -ffpe-trap=invalid,zero,overflow -fbounds-check -finit-real=nan -finit-integer=9999999 -finit-logical=true -finit-character=35
+FFLAGS_DEBUG = -O0 -g -fbacktrace -fno-fast-math -ffree-line-length-none -fno-backslash -pedantic -Waliasing -Wampersand -Wline-truncation -Wsurprising -Wtabs -Wunderflow -fdump-core -ffpe-trap=invalid,zero,overflow -fbounds-check -finit-real=nan -finit-integer=9999999 -finit-character=35
 
 TRANSCENDENTALS := -fast-transcendentals
 FFLAGS_OPENMP = -fopenmp

It is not clear if this is something we should be worried about, since this restart reproducibility test passes without this flag (including in repro mode, which is the mode in which we do all of our production runs). For what it is worth, this flag is also not included in the debug mode flags of SHiELD (see the flags in the template here).

Another notable aspect of this issue is that this only happens for simulations started from GFS initial conditions (e.g. the default.yml config file); for the restart.yml config, this restart reproducibility test always passes (whether we run in debug mode or not, whether we use the -finit-logical=true flag in debug mode or not).