E3SM-MMF run aborted with exit code 11 (segmentation fault) on an unsupported machine

[lw2304]

Dear E3SM team,

I'm attempting to run E3SM-MMF with momentum feedback using the branch
whannah/incite2021/momentum-transport-updated on an unsupported machine.

After customizing config_machines.xml and config_compilers.xml, I was able to successfully build the model.
However, when I submitted the run job to the SLURM system, the job aborted within a few seconds.
The only message in the log file is as follows:

Running job in: /data1/elzd_2023_00034/lw/lw_E3SM/cases/E3SM_ne30pg2_ne30pg2_F-MMF1_test_v2/run

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 14347 RUNNING AT e2311r5n4
=   EXIT CODE: 11
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 14362 RUNNING AT e2311r5n4
=   EXIT CODE: 11
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
   Intel(R) MPI Library troubleshooting guide:
      https://software.intel.com/node/561764
===================================================================================

There is no additional output from any component model logs (no atm.log, lnd.log, etc.).

Software environment:
Branch: whannah/incite2021/momentum-transport-updated
Compiler: intel/2021.3.0
MPI: intelmpi/2021.3.0
NetCDF: 4.7.4

According to Intel MPI’s documentation, exit code 11 corresponds to a segmentation fault, which may be caused by out-of-memory.
However, even after switching to a large-memory partition (512 GB per node), the same error persists.

Could this be related to an MPI runtime issue or something going wrong during model initialization on unsupported machines?
Any suggestion on how to enable more detailed debugging or logging output would be greatly appreciated.

Thank you very much for your help!

[whannah1]

first question - why are you using such an old branch? There were many MMF specific important updates since 2021, such as the variance transport scheme that official added to the master branch in 2022 that fixed a concerning checkerboard noise issue (see papers here and here). After that most fixes were not critical bug fixes since we diverted the development effort to other parts of the model, but there was still a good amount of code refactoring that happened. I would encourage you to switch to a modern branch. Even if you are specifically trying to reproduce an old experiment there is no capability in an old branch that is not preserved in the most recent master. Additionally, with the C++ version of the CRM (i.e. SAMxx) you will be able to run the CRM on GPUs for significant speed-up if your machine offers that.

second question - what CRM configuration are you wanting to run? 2D vs 3D? The 2D CRM can represent momentum feedback via the "ESMT" scheme (see paper here). If I remember correctly, I created that old INCITE branch to do a special run that used the ne45pg2 grid along with a large 3D CRM and explicit momentum feedback. If you are wanting to do a 3D experiment there's a nuanced caveat that has to do with the application of surface friction that you should know about. You can reach out to me on slack for more details.

While we don't officially support setting the model up on new machines I can provide some pointers.

• first of all - don't run your initial tests with ne30pg2 - use a smaller grid that only needs a single node - like ne4pg2.
• enable core file dumps with ulimit -c unlimited or whatever the equivalent is for your system.
• build with "debug mode" enabled by issuing this command prior to the build => ./xmlchange --file env_build.xml --id DEBUG --val TRUE

Since I was one of the main users and devs of E3SM-MMF in its heyday I'm happy to spend a bit more effort to ensure it is still being used. I can provide quicker feedback if you find me on the E3SM slack.

[lw2304]

Hi Walter,

I wanted to give you a quick update — the segmentation fault issue I mentioned earlier has been resolved. It turned out to be related to the Intel compiler linking against an older C++ standard library that wasn’t fully C++17-compliant, which caused Kokkos to fail. I also found that loading a newer GCC version (via module load compiler/gcc/9.3.0) could resolve the Kokkos compilation issue by providing a more up-to-date C++ standard library. However, when using this setup, the model crashed at runtime, likely because the environment contained both the Intel and GNU compiler toolchains, which caused conflicts between their libraries. Switching entirely to the GNU compiler toolchain eliminated these conflicts and allowed the model to run successfully.

However, the 3D MMF case still runs quite slowly. I remember you mentioned that SAMxx can run the CRM on GPUs to greatly accelerate the simulation. I’d like to learn more about that — specifically, what kind of hardware setup is required and whether there are any additional build or runtime configurations needed to enable GPU acceleration for the CRM.

Thanks for your help and guidance!

[whannah1]

Glad to hear you've overcome the CPU build issues - and yes, the 3D configuration can be very slow. Running the CRM on GPUs can be a huge boost. For typical 2D cases with a 64x1km CRM I've seen throughput over 15 simulated years per day (sypd) when running on GPUs at NERSC.

Despite the advantage of the GPU hardware acceleration, the overall throughput can still vary a lot depending on the number of nodes, for a variety of reasons. The MMF presents a very "GPU friendly" computational problem because aside from the data transfers at the start and end of each CRM call, the work stays on the GPU. This also means that you get relatively more GPU hardware acceleration as the workload of the CRM becomes larger. If you simply increase the size of the CRM on a fixed number of nodes w/ GPUs you tend to still see a bit of a slow-down, but the overall throughput is usually so much faster compared to running on the CPU that I'm not bothered. If you only have access to a small number of nodes then you might find yourself in a weird situation where the larger number of CPU nodes is actually faster. So the advantage of GPUs can differ case to case.

Unfortunately, the machine setup for GPUs has all the same challenges as a CPU configuration, plus a bit more. Aside from needing t provide a CUDA library and making sure the other libraries will work well with that, I think there are a few CMake changes required. It will probably feel like you're setting up another machine from scratch. Do you already have access to a machine with GPUs?

[lw2304]

I tested my cluster’s GPU setup and confirmed that the available GPUs are AMD DCUs using the ROCm platform. The ROCm driver and hardware are working properly (two GPUs per node).

From what I understand, SAMxx’s GPU acceleration currently relies on CUDA, which would mean it only supports NVIDIA GPUs. Could you please confirm if SAMxx can be built or executed on AMD GPUs using ROCm/HIP, or if CUDA-compatible hardware is required?

[whannah1]

I think we've run on AMD's in the past, but this is outside my area of expertise. Since the backend is now effectively handled by kokkos I think we have everything you need to get this working - but I can't really help with those setup details.

[lw2304]

Understood. Thank you very much for all your guidance and detailed responses over the past week — I truly appreciate the time and effort you’ve put into helping me resolve these issues. Your explanations have been both insightful and encouraging.

Best regards

[rljacob]

To verify that your machine was ported to E3SM correctly, try a simpler case like --compset X --res f19_g16.

[rljacob]

nodes:20 means it is set up to use 20 nodes. To double that, you would need to double the number of tasks. You can do that with "./xmlchange NTASKS=2560". Then do "./case.setup --reset" and "./case.build" and submit again.

[lw2304]

Oh, I see — I misunderstood you earlier. I did run the model on 20 nodes.
After checking the core dump, it seems the issue might be related to the standard C library, as I discussed with Walter. I’ll try compiling and running with the GNU compiler instead.
Thank you for your time and help.

[whannah1]

that reminds me - I recall at some point needing to change the version of the standard library with a flag like -std=c11 - do you know which version you are using?

[lw2304]

for C compiler, I set -std=c99.
for C++ compiler, I set -std=c++17.
Because the documentation of kokkos says it requires C++ compiler supports C++17 or later.

[whannah1]

🤔