HEMCO restart tests fail

[fvitt]

What happened?

When HEMCO emissions are used restarts are not bit-for-bit the same as continuous runs.
Regression testing give this failure:
FAIL ERP_Ln9_Vnuopc.f09_f09_mg17.FCSD_HCO.cheyenne_intel.cam-outfrq9s COMPARE_base_rest

Until this is fixed, HEMCO configurations will not be supported.

What are the steps to reproduce the bug?

ERP_Ln9_Vnuopc.f09_f09_mg17.FCSD_HCO.cheyenne_intel.cam-outfrq9s COMPARE_base_rest

What CAM tag were you using?

cam6_3_118

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

No response

Will you be addressing this bug yourself?

No

Extra info

@jimmielin will need to provide hemco external updates to address this reproducibility / restart issue.

[lizziel]

Following up about the HEMCO restart issue, I figured out the problem is vertical regridding of 3D emissions using instantaneous surface pressure. The specific problem is that the minimum read frequency for HEMCO is 1 hour on the hour, i.e. 0:00, 1:00, etc., but restarted runs which use a sub-hour timestep, e.g. 30 min, begin off the hour, i.e. 0:00 first run (okay), but then 3:30, 6:30, etc. I am able to eliminate differences by changing the timestep to 1 hour. This results in vertical regridding using identical pressures across all runs since no regridding occurs off the hour in any of the runs.

Solutions within HEMCO could include storage of raw file data within HEMCO to be regridded each timestep, or sub-hour reads. Neither are great from a performance point-of-view. Another option, which is simpler but would impact you, is having restarted runs begin on the hour rather than the first timestep after the hour. This is what we do in offline GEOS-Chem and within GEOS and hence we have never had this problem before. Note that all of my testing using CAM-chem not GEOS-Chem. Is there a reason you start the subsequent runs at the timestep after the time corresponding to the restart file (cam.r.*)?

To be more clear about how runs are currently split up in time within CAM, here is a breakdown of my own 30 min timestep runs.

Compset: FCHIST_HCO
Config params: REST_N=3, STOP_N=3, STOP_OPTION=nhours, ATM_NCPL=48

Run A – seven full timesteps (in GEOS-Chem CTM, HEMCO standalone, and GEOS this would only be six full timesteps)
nstep 0 - 00:00. <- HEMCO 3D read/regrid
nstep 1 – 00:30
nstep 2 – 01:00. <- HEMCO 3D read/regrid
nstep 3 – 01:30
nstep 4 – 02:00. <- HEMCO 3D read/regrid
nstep 5 – 02:30
nstep 6 – 03:00 <- HEMCO 3D read/regrid
nstep 7 – 03:30 (partial)

Run B (restarted) – six full timesteps
nstep 7 – 03:30. <- HEMCO 3D read/regrid
nstep 8 – 04:00. <- HEMCO 3D read/regrid
nstep 9 – 04:30
nstep10 – 05:00 <- HEMCO 3D read/regrid
nstep 11 – 05:30
nstep 12 – 06:00 <- HEMCO 3D read/regrid
nstep 13 – 06:30 (partial)

All additional subsequent runs are also six full timesteps. Why does the initial run have seven? Using six would solve the HEMCO problem. It is also more intuitive for a 3 hour run with 30 minute time intervals to execute 6 full timesteps.

[lizziel]

Another solution could be to store the last hour's edge pressures in the restart file and use those for the initial regrid upon restart. A cleaner solution, however, is to start runs that are restarted at integer number of hours on the hour rather than offset.

[adamrher]

That's great news that you tracked down this issue, @lizziel.

@brian-eaton I don't remember why the model integrates for a half time-step past STOP_N. Can you refresh my memory? Thanks!

[lizziel]

In case it helps, here is what the execution would look like to fix the problem:

Run A – six full timesteps
nstep 0 - 00:00 <- HEMCO 3D read/regrid
nstep 1 – 00:30
nstep 2 – 01:00 <- HEMCO 3D read/regrid
nstep 3 – 01:30
nstep 4 – 02:00 <- HEMCO 3D read/regrid
nstep 5 – 02:30
nstep 6 – 03:00 (partial)

Run B (restarted) – six full timesteps
nstep 6 – 03:00 <- HEMCO 3D read/regrid
nstep 7 – 03:30
nstep 8 – 04:00 <- HEMCO 3D read/regrid
nstep 9 – 04:30
nstep10 – 05:00 <- HEMCO 3D read/regrid
nstep 11 – 05:30
nstep 12 – 06:00 (partial)

etc.

[brian-eaton]

The ERP_Ln9_* test does an initial run of 5 steps and then does a 4 step restart. For a model configuration with a 1/2 hour timestep, that means the model is restarting at 9000 seconds (you can see this in the timestamp of the restart file) and so HEMCO is initializing to a different state for timestep 6 than it would have if it were doing a continuous 6 timestep run. This clearly won't pass the restart test.

The fact that CAM's timeloop does some calculations (and writes a confusing message about it to the log file) after the restart file is written and before control returns to the coupler is not relevant to this restart failure. The calculations that are done after the restart file is written are redone during the initialization phase of a restart run (don't ask why:).

CAM has historically required being able to restart from an arbitrary timestep. The benefit of that is mainly for debugging purposes. It sounds like HEMCO could be made to restore its state at a non-hour boundary if it
saved the edge pressures from the correct hour boundary to the restart file as suggested by Lizzie. I think this is the best solution if we want to keep the requirement that CAM be able to restart from an arbitrary timestep.

In the meantime I think we should adjust the restart tests with HEMCO to only restart on an hour boundary. I don't know the cime testing infrastructure well enough to know the best way to do this, but I assume it's not hard.

One thing keep on the radar though is that the following tests seem to randomly fail baseline comparisions, and this is not related to restarts:
SMS_Ld1.f09_f09_mg17.FCHIST_GC.derecho_intel.cam-outfrq1d (Overall: DIFF)
SMS_Lh12.f09_f09_mg17.FCSD_HCO.derecho_intel.cam-outfrq3h (Overall: DIFF)

[adamrher]

It sounds like HEMCO could be made to restore its state at a non-hour boundary if it
saved the edge pressures from the correct hour boundary to the restart file as suggested by Lizzie. I think this is the best solution if we want to keep the requirement that CAM be able to restart from an arbitrary timestep.

From a practical and user-friendly perspective, this seems like a requirement we should keep! We don't need the added hassle of making sure STOP_N lands on the hour for any arbitrary ATM_NCPL, especially if this becomes the default emissions driver in CAM7. Thanks for the clarification Brian.

[cacraigucar]

@brian-eaton - I would suggest we NOT change the HEMCO test so that it uses a setup which passes. Since HEMCO is targeted to be a default setup in CESM, I want to make sure we remember it is broken until it is fixed. I know it is a pain to have a continuously failing regression test, but in this case, I think it is important to keep it failing until a fix is provided for HEMCO.