Model Instability when Attempting to Use Custom Land Cover Inputs with Noah-MP 4.0.1

[mcthreems]

I attempted to run a simulation using a customized land cover input file, based on the igbp.bin file included in the distribution. My input is based on the MODIS land cover data for 2013. I was unable to find the MODIS files that match the built-in MODIS option available in the model, so tried this as an alternative approach. The LDT processing of the customized file occurs without errors, but when running LIS using the LDT output the model becomes unstable after ~8 months and crashes. There is no error message or acknowledgement of the crash in the log file, however the terminal output does print an mpi error (copied below). I have attached here the ldt config, lis config, land cover input file, and log file. My main question is whether there's an obvious issue with the custom input file, and if so how should I correct it?
discussion_zip.zip

MPI Error Message:
emitted longwave <0; skin T may be wrong due to inconsistent
input of SHDFAC with LAI
1 1 SHDFAC= 0.656851172 VAI= 3.49986291 TV= 332.304230 TG= 418.069336
LWDN= 324.148468 FIRA= -2471.71777 SNOWH= 0.00000000
STOP in Noah-MP
Note: The following floating-point exceptions are signalling: IEEE_INVALID_FLAG IEEE_DIVIDE_BY_ZERO

mpirun has exited due to process rank 1 with PID 0 on
node mz-dtn exiting improperly. There are three reasons this could occur:

1. this process did not call "init" before exiting, but others in
   the job did. This can cause a job to hang indefinitely while it waits
   for all processes to call "init". By rule, if one process calls "init",
   then ALL processes must call "init" prior to termination.

2. this process called "init", but exited without calling "finalize".
   By rule, all processes that call "init" MUST call "finalize" prior to
   exiting or it will be considered an "abnormal termination"

3. this process called "MPI_Abort" or "orte_abort" and the mca parameter
   orte_create_session_dirs is set to false. In this case, the run-time cannot
   detect that the abort call was an abnormal termination. Hence, the only
   error message you will receive is this one.

This may have caused other processes in the application to be
terminated by signals sent by mpirun (as reported here).

You can avoid this message by specifying -quiet on the mpirun command line.

[dmocko]

Hi @mcthreems

Thanks for your patience with a reply. Several LIS team members were on leave in July/August, we have lost some staff,
and we had important annual meetings this month.

I'm not sure why are you getting this error after ~8 months of simulation. Did you check all of your lislog files for a message?

Again, your configure.lis file might help as well. It's also possible that your simulation has run out of memory and crashed.

[mcthreems]

Thanks for getting back on this. I reran this simulation to double check it still produced the same issue, and it appears to do so. I have attached the log output, as well as the configure.lis file. Is there some way to determine if it ran out of memory? I don't see anything indicating that but may not be looking in the right place.
log_out.txt
configure.lis.txt

[dmocko]

Usually, SLURM or PBS will have a line included in your job standard output showing how much memory you used.

I don't see anything suspect in your configure.lis. Again, did you check ALL of your lislog files (not just the .0000 file)?

I will try to run and re-create your error, but won't be able to do it until next week.

[mcthreems]

I ran this using only 1 processor, so I had assumed that meant there would only be 1 log file. For similar reasons this wasn't run through a job scheduler like slurm so I need to check the memory usage while it runs. I appreciate you offering to try running it on your end.

[dmocko]

Yes, only 1 processor means just 1 log file. Thanks for letting me know.

[dmocko]

Hi @mcthreems

I had a chance to run LDT and LIS myself with your config files and your modis binary custom land cover file.

LIS ran without crashing for me. It's possible that one of your MERRA-2 metforcing files is corrupted.

Can you check the size/contents of the 20130818 files (slv, rad, lfo, and flx)?