Missing header file CD_GeoCoarsener.H during compilation

[Sduwzy]

Hello,

I’m trying to reproduce the CO₂ discharge simulation results using the source code provided in this repository. However, I encountered the following compilation error:

program.cpp:2:10: fatal error: CD_GeoCoarsener.H: No such file or directory
    2 | #include <CD_GeoCoarsener.H>
      |          ^~~~~~~~~~~~~~~~~~~
compilation terminated.

t seems that the file CD_GeoCoarsener.H is missing — I couldn’t find it in the repository or its submodules.

Could you please advise how to resolve this issue?

Is this header located in another branch or submodule that I might have missed?

Or has it been replaced by another file/class in the latest version?

Thank you very much for your help!

Best regards,

[rmrsk]

GeoCoarsener was removed recently, and you are probably building against a newer version of chombo-discharge.

Either revert to the git hash given in the README file, or update your program.cpp file in order to use the newest chombo-discharge features.

[Sduwzy]

Thank you for your kind guidance!
After rolling back to the previous version, I’ve successfully run the program.

I have one more question — could you please let me know approximately how much memory (RAM) is required when running your simulation?

Thank you again for your help!

[rmrsk]

No clue - probably a lot since some of those calculations ran on up to 10k CPU cores

[Sduwzy]

Hello,

Thank you very much for your previous help.

I have recently tried running the program using more than 10,000 CPU cores (with a total memory of about 50 TB). Each node has 64 processors and 512 GB of memory.

However, the job stopped because of a memory overflow, and several MPI processes were killed by the system.

Could you please kindly advise how this issue might be resolved?
For example, are there recommended settings or strategies (such as domain decomposition, grid configuration, or memory-related parameters) that could help reduce memory usage?

I sincerely appreciate your time and guidance.

Best regards,

[rmrsk]

Its hard for me to tell whats going on here. Can you paste your pout.0 file here, and your Make.defs.local. I'd like to see your .inputs file, too.

I can't help further without knowing at least a little bit about what is going on.

[Sduwzy]

HPCtest.zip
Thank you very much for your kind help!
Please find my .out and make file attached for your reference and further analysis.

[rmrsk]

1. The biggest issue is that you're not actually using MPI. You've indicated the compiler, but you've set MPI=FALSE in the Make.defs.local, so it will compile for serial execution without any support for distributed calculations.

2. From the error file it looks like you're running many instances of the same program, and they all break because there's no domain decomposition during the initial mesh generation step. I'm assuming that you want parallel calculations, so you should set MPI=TRUE in Make.defs.local.

3. These simulations were also done in 3D, so you must set DIM=3. Otherwise it will be planar 2D calculations.

4. You've also set USE_HDF = FALSE, which will turn off HDF5 output. Set it to TRUE if you want plot files.

[Sduwzy]

Hello,

Thank you very much for your guidance earlier. I followed your suggestions, modified the files, and after debugging the code now runs and produces the various .out logs.

However, when running with 3,200 cores for about 24 hours, I still don’t get any plt (plot/checkpoint) outputs—only the .out files are generated. I’ve attached several .out files, plus .slurm, and the relevant configuration files for your reference.

Could you please advise on possible causes or solutions?

HPC.zip

[rmrsk]

This indicates to me that you are linking incorrectly to HDF5.

First, in the slurm file the path is to an OpenMPI-version of HDF5, but the slurm file uses the same path to serial and parallel versions of HDF5. Please ensure that the path is correct. You also do not need to run for 24 hours -- you can set Driver.max_steps=0 while you debug/set up. An HDF5 file should show up at the first time step.

Second, try to run a clean rebuild: From the code root directory:

make -s -j12 pristine
make -s -j12 lib

Occasionally, Chombo doesn't rebuild correctly, and it sometimes helps to do a "fresh" setup when changing build options.

[Sduwzy]

Hello, and thank you very much for your quick response.

Regarding the “slurm file” you mentioned — did you mean the makerule (Make.defs) file?
In my makerule file, I did set the same path for both the serial and parallel versions of HDF5. However, when I checked the makerule provided in the chombo-discharge repository for Betzy
(https://github.com/chombo-discharge/chombo-discharge/blob/main/Lib/Local/betzy/Make.defs.MPI.HDF5.GNU
,shown in attached figure

),
it also appears to use the same path for both. Since this file is intended for running on the Betzy system, I assumed that using identical paths is acceptable.

Does this imply that when running with mpirun, the serial HDF5 flags are effectively ignored, so having the same path should not cause a problem?

In addition, I have already performed both
make -s -j12 pristine
make -s -j12 lib
when switching environments, so it seems the issue may not be caused by stale build files.

If possible, could you also share the makerule (Make.defs) and slurm submission script that you use on Betzy? They would be extremely helpful for comparison.

Thank you again for your guidance, and I would appreciate any further suggestions.

[rmrsk]

Its not about slurm - somethingabout HDF5 is not getting exposed (or you've set the plot interval to <= 0).

Using identical paths is fine, but you still need to point to an MPI compliant HDF build.

To debug, can you add a preprocessor error behind the HDF ifdef guards? In CD_Driver.H, add something like

#ifdef CH_USE_HDF5
#error "Compiling with HDF5"
#endif

If this compiles, it means that something has gone wrong and HDF5 is not getting exposed the way it should.

[Sduwzy]

Hello,

I added the preprocessor test as you suggested, and the compilation indeed stopped with:

/dssg/home/acct-weizy/weizy/chombo-discharge/Source/Driver/CD_Driver.H:16:2: error: #error "Compiling with HDF5"
   16 | #error "Compiling with HDF5"
      |  ^~~~~
make[1]: *** [/dssg/home/acct-weizy/weizy/chombo-discharge/Submodules/Chombo-3.3/lib/mk/Make.rules:517: o/3d.Linux.64.mpic++.gfortran.OPTHIGH.MPI/CD_Driver.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make: *** [GNUmakefile:12: discharge-source] Error 2

This indicates that CH_USE_HDF5 is being detected correctly, so HDF5 is exposed during compilation.
Given this result, it seems the issue may not be caused by HDF5 being disabled.

Do you have any suggestions on what I should check next?
For example:

• could it still be related to the specific HDF5 MPI build or linking setup?
• or could plot/checkpoint output be failing for another reason even though HDF5 is enabled?

Thank you again for your help — I really appreciate your guidance.

Best regards,

[rmrsk]

I dont know whats going on here - my only guess is that the HDF5 installation is broken. Btw, I'd like you to make sure that you are linking to an MPI-compliant HDF build. In the Betzy configuration file both paths are to MPI versions (it looks like your paths are to a serial version).