Failing compare `ERP_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-L72`

[ndkeen]

At some point, this test started failing. On cdash, the report is a test named
ERP_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-bfbhash--eamxx-L72

but I verified we dont need the eamxx-bfbhash modifier. Note the eamxx-L72 is still needed and the test has always been 72 levels (we recently changed default to 128 levels, so now need the modifier).

This test fails compare:

ERP_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-L72

/pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/c29-oct25/ERP_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-L72.gh7843

/pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/c29-oct25/ERP_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-bfbhash--eamxx-L72.gh7843

However, interesting, these pass:

PEM_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-L72
ERS_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-L72
PET_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-bfbhash--eamxx-L72

suggesting it may not be simply an issue of change with MPI count, or restarts, but some combination.

I was also trying a DEBUG variety:

ERP_D_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-L72
ERS_D_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-L72

which both pass (note they need more than debug qos walltime, 48 and 40 minutes each)

I also tried with intel compiler. We don't really test eamxx much with intel compiler on pm-cpu and there is a known issue with tests not being BFB when varying MPI tasks (can point to older issues #6834 and #7746 ). So I didnt expect it to pass here and they do not -- these 2 fail compare:

ERP_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_intel.eamxx-L72
PEM_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_intel.eamxx-L72

Some other tests I tried:

fail compare  ERP_P1024x1_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-L72

The P1024x1 not needed as it defaults to 1 thread. But does show we still get fail with more nodes as default cases use 2 nodes and 1024 MPI's would use 8.

Note from TestStatus.log of one of the cases:

SUMMARY of cprnc:
 A total number of    413 fields were compared
          of which    149 had non-zero differences
               and      0 had differences in fill patterns
               and      0 had different dimension sizes
               and      0 had different data types
 A total number of      0 fields could not be analyzed
 A total number of      0 time-varying fields on file 1 were not found on file 2.
 A total number of      0 time-constant fields on file 1 were not found on file 2.
 A total number of      0 time-varying fields on file 2 were not found on file 1.
 A total number of      0 time-constant fields on file 2 were not found on file 1.
  diff_test: the two files seem to be DIFFERENT 

[ndkeen]

I wanted to experiment with turning on more hashing, so I began running a standard conus test from a launch script. First I tried 2,4,8 nodes (recall this is on CPU), no threading, and see that all 3 of those cases are BFB after 22 steps. OK, going back to ERP, we see that it actually does more than just change the MPI tasks and perform a restart -- it also adjusts threading. It will halve the number of threads being used, or if not threads, will still set FORCE_BULD_SMP=TRUE. Then I tried this in my standard conus cases and interestingly, begin to see diffs.

Here are 2 conus cases that both set FORCE_BULD_SMP=TRUE, and only differ in number of MPI tasks used.

perlmutter-login20% zgrep hash /pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/c29-oct25/conus.F2010-SCREAMv1-noAero.conusx4v1pg2_r05_oECv3.c29-oct25.gnu.n002.22s.bfb1.fst/run/e3sm.log.44804029.251103-100031.gz > c2
perlmutter-login20% zgrep hash /pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/c29-oct25/conus.F2010-SCREAMv1-noAero.conusx4v1pg2_r05_oECv3.c29-oct25.gnu.n004.22s.bfb1.fst/run/e3sm.log.44804030.251103-100031.gz > c4
perlmutter-login20% diff -w c2 c4 | head
261,262c261,262
<   0: hxxhash>              5 0 ef1a6691c0586ccb (T BE-post-ComposeTransport-q-HV-0)
<   0: hxxhash>              5 1 6cd25c85b4c2a361 (T BE-post-ComposeTransport-q-HV-0)
---
>   0: hxxhash>              5 0 ef1a6691c1148ac3 (T BE-post-ComposeTransport-q-HV-0)
>   0: hxxhash>              5 1 6cd25c85b57ec159 (T BE-post-ComposeTransport-q-HV-0)

I might expect PET_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-bfbhash--eamxx-L72 to then fail compare, but it does not.
PET_P512_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-bfbhash--eamxx-L72
is also passing.

I also tried an even simpler (non-conus) case (--res ne30pg2_ne30pg2 --compset F2010-SCREAMv1), with the force on building with openmp, with both intel compiler and gnu compiler. I see the same conclusion: the 2 cases are different -- and the first bfbhash diff is in the same location as above.

perlmutter-login19% zgrep hash /pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/c29-oct25/f30.F2010-SCREAMv1.ne30pg2_ne30pg2.c29-oct25.intel.n002.22s.bfb1.fst/run/e3sm.log.44805872.251103-105250.gz > i2
perlmutter-login19% zgrep hash /pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/c29-oct25/f30.F2010-SCREAMv1.ne30pg2_ne30pg2.c29-oct25.intel.n004.22s.bfb1.fst/run/e3sm.log.44806439.251103-111522.gz > i4
perlmutter-login19% diff i2 i4 | head -6
261,262c261,262
<   0: hxxhash>              5 0 4c8e155b6b1d7f97 (T BE-post-ComposeTransport-q-HV-0)
<   0: hxxhash>              5 1 8b49ae1e23858e64 (T BE-post-ComposeTransport-q-HV-0)
---
>   0: hxxhash>              5 0 4c8e155b6b1d683e (T BE-post-ComposeTransport-q-HV-0)
>   0: hxxhash>              5 1 8b49ae1e2385770b (T BE-post-ComposeTransport-q-HV-0)

where I'm pointing out that the diff is happening in what looks like the same place.

[ndkeen]

I can reproduce the issue even with ne4 (--res ne4pg2_oQU480 --mach pm-cpu --compset F2010-SCREAMv1). This this case, running 5 days, with either 96 or 64 MPI's:

                                       BFB 64 vs 96 MPI's   
intel                                        no
intel, FORCE_SMP_BUILD=TRUE                  no

gnu                                          yes
gnu, FORCE_SMP_BUILD=TRUE                    no

Where ./xmlchange FORCE_SMP_BUILD=TRUE is an option that allows building with openmp flags, but only running with 1 thread -- this will not be BFB with serial cases, but should be BFB with other threaded cases.

I'm currently only looking at diff in the full hash output -- and again the difference in the full hash output is same place:

<  0: hxxhash>              5 0 49ee426bbdda92a4 (T BE-post-ComposeTransport-q-HV-0)
<  0: hxxhash>              5 1 5c924c885369eef2 (T BE-post-ComposeTransport-q-HV-0)
---
>  0: hxxhash>              5 0 49ee426bbdda6bce (T BE-post-ComposeTransport-q-HV-0)
>  0: hxxhash>              5 1 5c924c885369c81c (T BE-post-ComposeTransport-q-HV-0)

If this holds, we were just lucky to not see it with GNU builds.

I also just tested PACK_SIZE=1, and while results are different (than default PACK_SIZE=16), I still see the same issue as above -- a diff comparing ne4 96 vs 64 MPI's in same location.

Verified same issue with 72 vertical levels.
And also verified the difference happens before the 1st step is completed, ie even ne4, 72 levels, with 1 step can reproduce.

[ndkeen]

Going back thru older hashes to see when this fail started to happen, it looks to be #7740 which is my PR to update modules. I'll have to figure out what changed. First guess is gcc compiler and/or cray-libsci. These are all the default/latest versions.

I verified that for a checkout of hash 29b14d892a (ie the one introducing PR7740), if I change:

gcc-native/13.2  cray-libsci/24.07.0
back to what we had before:
gcc-native/12.3  cray-libsci/23.12.5

the test passes. Will see if it's one or the other -- yep verified it's only the compiler change (libsci not important).
The exact test I was using here:

ERP_Ln22.conusx4v1pg2_r05_oECv3.F2010-SCREAMv1-noAero.pm-cpu_gnu.eamxx-bfbhash--eamxx-L72

Asking about differences between these two GCC version and how it might relate to single-thread openmp:

GCC 13 includes continued improvements and fixes to ensure full compliance with the OpenMP standard,
particularly OpenMP 5.0 and 5.1.

Implicit Scoping Changes: The rules for implicit data sharing (when a variable is not explicitly declared shared, private, etc.)
have been fine-tuned for various constructs. If your code relies on a variable having a certain default scope (e.g.,
firstprivate) that was incorrectly inferred by GCC 12 but is now correctly inferred by the stricter GCC 13, the sequential
behavior might change.

Default device_type on Host: Changes in how the compiler handles the OpenMP device model, even when targeting
the host $CPU, can affect initialization. If any OpenMP construct uses device-related clauses, the newer
compiler's implementation might introduce subtle runtime differences.

Initialization of Firstprivate Variables: Stricter rules regarding the initialization of variables that are made firstprivate
might reveal an underlying bug in your code. For example, if GCC 12 deferred initialization in a way that hid a dependency,
GCC 13 might enforce it immediately, changing the single-threaded result.

So this might be something where we have some openmp-related assumptions no longer true? And Intel was trying to warn us.

[ndkeen]

Just to follow-up with another suggestion. I verified that I'm seeing different results even when using 2 threads -- ie comparing 2 threads with 1 MPI and 2 threads with 2 MPI's. I also made sure I get same conclusions with ne4pg2_ne4pg2 (as I was prev using ne4pg2_oQU480.

in /pscratch/sd/n/ndk/e3sm_scratch/pm-cpu/c29-oct25
comparing a case using 2 threads (1 and 2 MPIs)
f4.F2010-SCREAMv1.ne4pg2_oQU480.c29-oct25.gnu.n001.p001x211111.18s.bfb2.fst.L72/
f4.F2010-SCREAMv1.ne4pg2_oQU480.c29-oct25.gnu.n001.p002x211111.18s.bfb2.fst.L72/

and with ne4pg2_ne4pg2, first comapring 1/2 MPI's with 1 thread, then 1/2 MPI's with 2 threads
f4.F2010-SCREAMv1.ne4pg2_ne4pg2.c29-oct25.gnu.n001.p001x211111.18s.bfb2.fst.L72/
f4.F2010-SCREAMv1.ne4pg2_ne4pg2.c29-oct25.gnu.n001.p002x211111.18s.bfb2.fst.L72/

f4.F2010-SCREAMv1.ne4pg2_ne4pg2.c29-oct25.gnu.n001.p001x111111.18s.bfb2.fst.L72/
f4.F2010-SCREAMv1.ne4pg2_ne4pg2.c29-oct25.gnu.n001.p002x111111.18s.bfb2.fst.L72/

looking at hashes in e3sm.log files with a script that shows the first divergence, and then the first divergence after the first step (which is always simply the next bfbhash)

=================================================================================
REPORT: ABSOLUTE FIRST HASH DIFFERENCE (Starting at First Step)
================================================================================
#  DIVERGENCE FOUND at BFB Step 0 (Entry #504)

--- Context (Last 3 identical hashes) ---
 0: hxxhash>              5 0 58d0e91b004dfa9e (E BE-pre-ComposeTransport-q-HV-0)
 0: hxxhash>              5 1 c9f0a2ef99fc211b (E BE-pre-ComposeTransport-q-HV-0)
 0: hxxhash>              5 2 b874c377ba59c4a5 (E BE-pre-ComposeTransport-q-HV-0)

--- Divergence Found ---
-0: hxxhash>              5 0 4912c99cc6528345 (T BE-pre-ComposeTransport-q-HV-0)  (File 1)
+0: hxxhash>              5 0 4912c99cc6513141 (T BE-pre-ComposeTransport-q-HV-0)  (File 2)
================================================================================

================================================================================
REPORT: FIRST DIVERGENCE AFTER INITIALIZATION (Skipping First Step)
================================================================================
#  DIVERGENCE FOUND at BFB Step 1 (Entry #626)

--- Context (Last 3 identical hashes) ---
 0: hxxhash>              5 2 b874c377ba59c4a5 (E BE-post-ComposeTransport-qdp-DSS-1)
 0: hxxhash>              5 0 306d4c2831441b30 (T BE-post-ComposeTransport-qdp-DSS-1)
 0: hxxhash>              5 1 cc2586cf50400a3c (T BE-post-ComposeTransport-qdp-DSS-1)

--- Divergence Found ---
-0: bfbhash>              1 4ec4713471b2e5cc (Hommexx)  (File 1)
+0: bfbhash>              1 4ec4713481d9cef3 (Hommexx)  (File 2)
================================================================================