GCHP abort with status -124 when writing checkpoints (pnc4) in parallel with multiple writers

[Daisy0419]

Your name

Daisy Wang

Your affiliation

Washington University in Saint Louis

Please provide a clear and concise description of your question or discussion topic.

When running GCHP with Midrun_Checkpoint = ON and multiple writers (6), the model writes the first checkpoint file then aborts with MAPL status = -124.
That first checkpoint file has the “correct” size but appears to be all zeros while a single-writer run on the same setup produces a valid netCDF-4 checkpoint.

This looks like a invalid parallel write via Parallel-NetCDF (or format mismatch), followed by a read that returns -124.

The same problem appears for both versions 14.5.2 and 14.6.3 on both AWS + Lustre (FSx) and compute1.

Question:

• Has anyone encountered same issue before? What are the possible causes?
• Could you please suggest how do I further debug or fix the issue?

Environment:
Environment (build from source)
MPI: Intel MPI 2021.13
Compilers: GCC/GFortran
HDF5 (parallel): 1.14.6
PnetCDF: 1.14.1
netCDF-C: 4.9.3
netCDF-Fortran: 4.6.2
PIO: 2.6.6
ESMF: 8.4.2

Error Log:

                                                             Mem/Swap Used (MB) at HISTMAPL_GenericInitialize=  5.277E+04  0.000E+00
                                                          Mem/Swap Used (MB) at EXTDATAMAPL_GenericInitialize=  5.246E+04  0.000E+00
                                                                      Mem/Swap Used (MB) at MAPL_Cap:TimeLoop=  5.128E+04  0.000E+00
 Character Resource Parameter: GCHPchem_INTERNAL_CHECKPOINT_TYPE:pnc4
 Using parallel NetCDF for file: Restarts/gcchem_internal_checkpoint.20190701_0000z.nc4
pe=00072 FAIL at line=00325    NetCDF4_FileFormatter.F90                <status=-124>
pe=00072 FAIL at line=03841    NCIO.F90                                 <status=-124>
pe=00096 FAIL at line=00325    NetCDF4_FileFormatter.F90                <status=-124>
pe=00096 FAIL at line=03841    NCIO.F90                                 <status=-124>
pe=00072 FAIL at line=04081    NCIO.F90                                 <status=-124>
pe=00072 FAIL at line=05807    MAPL_Generic.F90                         <status=-124>
pe=00072 FAIL at line=02472    MAPL_Generic.F90                         <status=-124>
pe=00096 FAIL at line=04081    NCIO.F90                                 <status=-124>
pe=00096 FAIL at line=05807    MAPL_Generic.F90                         <status=-124>
pe=00096 FAIL at line=02472    MAPL_Generic.F90                         <status=-124>
pe=00096 FAIL at line=02387    MAPL_Generic.F90                         <status=-124>
pe=00096 FAIL at line=01807    MAPL_Generic.F90                         <status=-124>
pe=00096 FAIL at line=02319    MAPL_Generic.F90                         <status=-124>
pe=00096 FAIL at line=01807    MAPL_Generic.F90                         <status=-124>
pe=00096 FAIL at line=01343    MAPL_CapGridComp.F90                     <status=-124>
pe=00096 FAIL at line=01300    MAPL_CapGridComp.F90                     <status=-124>
pe=00096 FAIL at line=01260    MAPL_CapGridComp.F90                     <status=-124>
pe=00096 FAIL at line=00837    MAPL_CapGridComp.F90                     <status=-124>
pe=00096 FAIL at line=00977    MAPL_CapGridComp.F90                     <status=-124>
pe=00096 FAIL at line=00313    MAPL_Cap.F90                             <status=-124>
pe=00096 FAIL at line=00258    MAPL_Cap.F90                             <status=-124>
pe=00096 FAIL at line=00192    MAPL_Cap.F90                             <status=-124>
pe=00096 FAIL at line=00169    MAPL_Cap.F90                             <status=-124>
pe=00096 FAIL at line=00029    GCHPctm.F90                              <status=-124>
Abort(-1602421760) on node 96 (rank 96 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -1602421760) - process 96

NetCDF Config:

nc-config --all

This netCDF 4.9.3 has been built with the following features: 

  --cc                -> mpicc
  --cflags            -> -I/usr/local/include -I/usr/local/include -I/usr/local/include -I/usr/local/include
  --libs              -> -L/usr/local/lib -lnetcdf
  --static            -> -lpnetcdf -lhdf5_hl -lhdf5 -lm -lz -lbz2 -lzstd -lxml2 -lcurl 
  --has-dap           -> yes
  --has-dap2          -> yes
  --has-dap4          -> yes
  --has-nc2           -> yes
  --has-nc4           -> yes
  --has-hdf5          -> yes
  --has-hdf4          -> no
  --has-logging       -> no
  --has-pnetcdf       -> yes
  --has-szlib         -> no
  --has-cdf5          -> yes
  --has-parallel4     -> yes
  --has-parallel      -> yes
  --has-nczarr        -> yes
  --has-zstd          -> yes
  --has-benchmarks    -> no
  --has-multifilters  -> yes
  --has-stdfilters    -> bz2 deflate zstd
  --has-quantize      -> yes

  --prefix            -> /usr/local
  --includedir        -> /usr/local/include
  --libdir            -> /usr/local/lib
  --plugindir         -> /usr/local/hdf5/lib/plugin
  --plugin-searchpath -> /usr/local/hdf5/lib/plugin:/usr/local/hdf5/lib/plugin
  --version           -> netCDF 4.9.3
  --build-system      -> autotools

ESMF log:

20251023 210156.405 INFO             PET026 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20251023 210156.405 INFO             PET026 !!! THE ESMF_LOG IS SET TO OUTPUT ALL LOG MESSAGES !!!
20251023 210156.405 INFO             PET026 !!!     THIS MAY CAUSE SLOWDOWN IN PERFORMANCE     !!!
20251023 210156.405 INFO             PET026 !!! FOR PRODUCTION RUNS, USE:                      !!!
20251023 210156.405 INFO             PET026 !!!                   ESMF_LOGKIND_Multi_On_Error  !!!
20251023 210156.405 INFO             PET026 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20251023 210156.405 INFO             PET026 Running with ESMF Version   : 8.4.2
20251023 210156.405 INFO             PET026 ESMF library build date/time: "Oct 23 2025" "17:14:39"
20251023 210156.405 INFO             PET026 ESMF library build location : /tmp/esmf-8.4.2
20251023 210156.405 INFO             PET026 ESMF_COMM                   : intelmpi
20251023 210156.478 INFO             PET026 ESMF_MOAB                   : enabled
20251023 210156.478 INFO             PET026 ESMF_LAPACK                 : enabled
20251023 210156.478 INFO             PET026 ESMF_NETCDF                 : enabled
20251023 210156.478 INFO             PET026 ESMF_PNETCDF                : enabled
20251023 210156.478 INFO             PET026 ESMF_PIO                    : enabled
20251023 210156.478 INFO             PET026 ESMF_YAMLCPP                : enabled
20251023 210156.678 INFO             PET008 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20251023 210156.678 INFO             PET008 !!!        MOAB turned OFF            !!!
20251023 210156.678 INFO             PET008 !!! Meshes now created using native   !!!
20251023 210156.678 INFO             PET008 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20251023 210157.478 ERROR            PET008 ESMF_Clock.F90:887 ESMF_ClockGetAlarm() Failure  - Internal subroutine call returned Error
            PET051 ESMF_MOAB                   : enabled
20251023 210156.531 INFO             PET051 ESMF_LAPACK                 : enabled
20251023 210156.531 INFO             PET051 ESMF_NETCDF                 : enabled
20251023 210156.531 INFO             PET051 ESMF_PNETCDF                : enabled
20251023 210156.531 INFO             PET051 ESMF_PIO                    : enabled
20251023 210156.531 INFO             PET051 ESMF_YAMLCPP                : enabled
20251023 210156.679 INFO             PET065 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20251023 210156.679 INFO             PET065 !!!        MOAB turned OFF            !!!
20251023 210156.679 INFO             PET065 !!! Meshes now created using native   !!!
20251023 210156.679 INFO             PET065 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
20251023 210157.479 ERROR            PET065 ESMF_Clock.F90:887 ESMF_ClockGetAlarm() Failure  - Internal subroutine call returned Error

[yantosca]

Tagging @yuanjianz @lizziel

[yuanjianz]

@Daisy0419 Can you also share your GCHP.rc and environment variables set in your run script? I don't think GCHP actually use pnetcdf at all. I think the parallel writing is written in MAPL.
Also, I never used multiple writers on Compute1, but this is necessary on AWS with EFA enabled.

[Daisy0419]

Hi @yuanjianz,

Yes—on AWS the instance has EFA enabled.
I’ve attached a zip with my run configurations:

• GCHP.rc
• setCommonRunSettings.sh
• run_gchp.sh

env_config.zip

Thanks for your help.

[yuanjianz]

Hi @Daisy0419,
Based on the files you provided, I have several concerns:

1. When using multiple writers, I always turn on WRITE_RESTART_BY_OSERVER: NO to WRITE_RESTART_BY_OSERVER: YES.
2. I have never used IntelMPI on AWS, but I my experience with EFA on AWS with OpenMPI suggest we should probably exclude OFI from BTL (I don't know how it is equivalent in IntelMPI). Please see GCHP HISTORY and restart/checkpoint issues on AWS with EFA enabled #493. The CMA issue is probably not related if your error is only related to checkpoint writing.

[Daisy0419]

Hi @yuanjianz ,

Thanks for pointing these out! After switching WRITE_RESTART_BY_OSERVER: YES, the run no longer aborts, but I’m seeing much slower I/O even with 6 writers:
One 1-day C48 silumation with hourly checkpoints takes ~997 s with single-writer + WRITE_RESTART_BY_OSERVER: NO, vs ~1456 s with six-writers + WRITE_RESTART_BY_OSERVER: YES.

Do you have any ides why is this the case? Are there any documentations explain the I/O behaviour with and without "WRITE_RESTART_BY_OSERVER ", and how the parallel I/O is done?

[yuanjianz]

@Daisy0419 Here is the MAPL wiki: https://github.com/GEOS-ESM/MAPL/wiki/PFIO:-a-High-Performance-Client-Server-I-O-Layer. You can find some related informations but GCHP's MAPL might not match the latest's version's behaviour.

I would recommend that if a single writer with no-o-server functions, then there is no need to do multi-writers with o-server. It is only useful when you have thousands of cores running and your restart file is very large. The overhead is big when the scale is small.

[lizziel]

Hi Daisy, note that we have in ReadTheDocs this note about number of writers and using the O-server: "Note that none of these settings are dependable ways to speed up or fix a run because they are dependent on file system and MPI stack." This is from the MAPL developers who implemented the O-server and we can try to get more information for specific setups that have worked well and which have not. The best method is to reduce the number of checkpoints written. Do you need hourly? Is it possible to put the information into a diagnostic instead?

[Daisy0419]

Hi Lizziel,

Thank you for the clarification. We don’t actually need hourly checkpoints, I was just testing the I/O time for checkpoint writing.

[lizziel]

Hi @Daisy0419, got it. Yes, writing restart files is very intensive. It is less so for simulations with fewer species.

[lizziel]

Hi @Daisy0419, let us know if there are any specific things related to checkpoint writing you would like to bring up with the GMAO MAPL developers.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If there are no updates within 7 days it will be closed. You can add the "never stale" tag to prevent the issue from closing this issue.