WARNING:CAAR: dp3d too small. dt_remap may be too large

[pgoddard08]

Hello,

I am running NARRM on E3SMv2.0. I've run for about 37 years, but I got an error this morning and I'm not sure how to proceed:
from the e3sm.log:

672:  WARNING:CAAR: dp3d too small. dt_remap may be too large
 675:  WARNING:CAAR: dp3d too small. dt_remap may be too large
 672:  k,dp3d(k), dp0:           72   36.4322683725957        300.712101647604
 675:  k,dp3d(k), dp0:           72   36.4322683725957        300.712101647604
 675:  WARNING:CAAR: dp3d too small. dt_remap may be too large
 675:  k,dp3d(k), dp0:           72   34.4200870207234        300.712101647604
 672:  WARNING:CAAR: dp3d too small. dt_remap may be too large
 672:  k,dp3d(k), dp0:           72   34.4200870207234        300.712101647604
 670:  WARNING:CAAR: theta<   100.000000000000       applying limiter
 670:  k,vtheta(k):           72   97.5093357302506

And this repeats until the model crashes.

It looks like a possible fix is to alter the timestep. If someone could verify that fix and how to alter the timestep, I would greatly appreciate it.

Many thanks,
Paul

[mahf708]

Hello, @ndkeen suggested I take a look at this issue.

This is a classic instability occurring, causing the dycore to abort as it can't recover.

There are several options to proceed, but they all carry risks. I think it would help us help to know more about the details of what you're trying to achieve and where:

1. What compset?
2. What resolution specifically?
3. What code version?
4. What machine?
5. What length is desired and how many different configurations/perturbations?

For fixes, you can:

1. Modify dycore settings lightly so that the trajectory changes a little
2. Introduce a small perturbation from the previous restart to alter the trajectory a little
3. Fundamentally reduce the timestep of the model (will make model slower, and may change the configuration significantly)

Let us know what you'd like to do (after answering the first 5 questions) and we can proceed

[pgoddard08]

Hi, thank you for your help.

1. The compset is WCYCL20TR (we began with restart files in 1950)
2. The resolution is northamericax4v1pg2_WC14to60E2r3
3. We are using the maint-2.0 tag of E3SM
4. We are running locally at Indiana University on a machine called Big Red 200 (https://servicenow.iu.edu/kb?id=kb_article_view&sysparm_article=KB0026317)
5. We would like to run up to year 2015 (currently crashed at 1987). Eventually, we will conduct several ensemble members of this Historical run and also run under SSP forcing for 2015-2100.

I'm not sure which you would recommend from the 3 fix options, but options 1 and 2 I believe will work for our needs.

Thanks,
Paul

[mahf708]

as @mt5555 said below, you can try: ./xmlchange BFBFLAG=FALSE as a first good (non-intrusive) option here. I will list a few other non-intrusive options that will likely help you get some more years.

Please take note of Mark's message: #7905 (comment), especially the last part:

The fact that it ran for 37 years suggests the timesteps are fine, and something else is wrong.

[mt5555]

Since this is a rare crash (after 37 years), the trajectory perturbation is a good thing to try. The tried-and-true way to do this is to go back at least a month or more before the crash, and restart after toggling the BFBFLAG (which introduces roundoff level changes):
./xmlchange BFBFLAG=FALSE
if that gets through the crash, you can turn it back on for the next restart.

if you have time and want do diagnose further, it would be good to post more details, such as where the crash occurs, and plots of the fields that are going bad and the fluxes. This looks like a problem at the surface - so could be the "coldT" instability, or a problem with the surface fluxes from the other components.

The error message ("dt_remap may be too large") is misleading. The fact that it ran for 37 years suggests the timesteps are fine, and something else is wrong.

[pgoddard08]

Thank you @mt5555 and @mahf708
I utilized ./xmlchange BFBFLAG=FALSE and restarted a few months prior to the crash. The simulation has continued through/past the point of the crash!

Would you recommend I stop the simulation and restart during a date past the previous crash with ./xmlchange BFBFLAG=TRUE ? I'm curious whether I should just continue running as is, or if it's important that I return to ./xmlchange BFBFLAG=TRUE

Thanks!
Paul

[ndkeen]

That's great that the easiest thing to try worked. I would recommend toggling it back to BFB (no need to cancel, just on next submission). Based on previous experience, I would expect you will run into similar issue with BFBFLAG=FALSE. So for sanity, you might prefer the default for your case to be BFBFLAG=TRUE, with occasional spans of BFBFLAG=FALSE.

[pgoddard08]

Will do, thank you all very much