proposed March 2025 experiments to progress identification of next ESM1.6 configuration

[har917]

@JhanSrbinovsky @pearseb @matt-csiro @rml599gh

To date (13/3/2025) the ESM1.6 (codebase, initial conditions, namelists and suites) is showing promise but also some issues. In particular there is not yet an ESM1.6 configuration comprising MOM5, CICE4, UM7 + CABLE3 that

• has satisfactory surface energy and water balance,
• has a satisfactory global ocean temperature drift,
• has a satisfactory global salinity drift,
• uses CABLE3 and a land cover distribution derived from LUH3,
• is able to run land-use change experiments, and
• satisfies run-to-run reproducibility and hence would be publishable.

The following experiments are suggested as a way of narrowing down the remaining problems.

1. a rerun of base-70b173e-dev-preindustrial+concentrations-253fc085 (Feb25) but where the cable namelist is correct (the default) from year 0. This is base-70b173e-dos-dev-preindustrial+concentrations-98ee924c
2. a run that uses the feature development feat-revised_canopy_epot-dev-preindustrial+concentrations-75ca640c (epot = head of UM7 + CABLE3, MOM5+WOMBAT-lite, the default cable namelist, and LUH3 derived land cover) but where the atmospheric/land initial conditions are taken from the latter part of the base-70b173e-dev-preindustrial+concentrations-253fc085 (Feb25) run.
3. substituting the feat-revised_canopy_epot-dev-preindustrial+concentrations-75ca640c (epot) executable and default namelist, and an initial condition from latter part of base-70b173e-dev-preindustrial+concentrations-253fc085 (Feb25) run - into the long run pi_concentrations-expt-c55f7217 (the long run managed by @pearseb, but which currently uses CABLE2.5).

The thought process behind this:

1. This will assess/confirm whether the origin of the ocean temperature drift in the latest runs are consistent across all runs that have the switch to LUH3 (feat-revised_canopy_epot-dev-preindustrial+concentrations-75ca640c uses LUH3 and had a stable ocean, but also had an incorrect namelist in the initial periods).
2. This will assess/confirm whether the initial conditions used in the LUH3 runs are a contributing factor. base-70b173e-dev-preindustrial+concentrations-253fc085 managed to 'get through' the initial period without triggering a temperature drift - so hopefully the use of an UM/CABLE initial condition from that run will avoid the drift seen in the epot run.
3. this would be a short cut to the next proto-type ESM1.6 - it would assess whether the ocean initial conditions are particularly sensitive to shocks from the land (e.g. freshwater flux).

For these - the primary challenge appears to be the desire to mix/match the initial conditions from existing runs (i.e. there's a need to alter time stamps) and possibly from non-standard locations, and to get payu to pick these up.

The ability to run land use change experiments likely needs further work in creating the LUH3 derived land cover maps. The ESM1.6-AMIP-historical configuration is able to work with CMIP6-LUH2 information but fails after 1 day with LUH3 derived maps so this is a separate issue for now.

---

Also - CABLE developers are also testing a further potential change which seeks to improve the agreement between the land radiation fluxes as seen by the UM diagnostics and the UM-land diagnostics (see UM7 ACCESS-NRI/JULES#293 ).

[JhanSrbinovsky]

Following ACCESS-NRI/UM7#4 the major changes to the proposed model are:

• default config now includes LUH3 PFT map

• default config now includes cable.nml switches/parameters
  - l_rev_corr = .TRUE.
  - snmin = 0.11

• major code deveioments are:
  - immediate developments following ACCESS-NRI/UM7#4
  - epot
  - initalizations

   - subsequent developments 
         - adjust water accumulation in lakes
         - exner halo
         - shuffle
  
   The latter are referred to as  subsequent developments because we spent quite a bit of time testing the immediate developments, and these developments occurred in the meantime. Initially they were tested prior to the default configs being updated as described above. Initially the "epot" modification seemed to result in an ocean temperature that was drifting downwards. It was surprising that this change should have such a dramatic impact. It was unclear what configs were applied and when. Further confusion arose when one of the simulations suggested that the epot change did NOT in fact cause the ocean temperature to trend downwards.
  

In the interest of starting a new run on sapphire rapids nodes, starting from ocean restarts at 842 years and using the latest ocean executable (pr31-3). And UM restarts at 142 years (from ACCESS-NRI/UM7#4) we have decided to start 2 runs, which are equivalent in (the latest default) configuration for both build and runtime, and two branches of source code "reflecting" the HEAD of UM7 dev-access-esm1.6 . This is including the immediate snd subsequent developments described above.

One simulation uses the HEAD of UM7 dev-access-esm1.6 - MarchEpot. One simulation uses a branch from HEAD of UM7 dev-access-esm1.6 where the ONLY change is reversal of the canopy%epot revision - MarchNOepot.

MarchEpot

UM7 branch:

ACCESS-ESM1.6 pr43-17:

PAYU link: /g/data/p66/jxs599/ESM16/PAYU/Dev/MarchEpot

MarchNOepot

UM7 branch:

ACCESS-ESM1.6 pr43-18:

PAYU link: /g/data/p66/jxs599/ESM16/PAYU/Dev/MarchNOepot

Note: The payu clone is a slightly modified adaptation of:
payu clone https://github.com/ACCESS-NRI/access-esm1.6-configs -B preindustrial-sapphire-rapids

Which I implemented on Friday night but I still have to run past @manodeep and/or @blimlim.

MarchEpot fell over after 33-34 years on Sunday night. The .err was a missing ocean restart. I restarted without perturbation. It seems to have fallen over in the same place although I haven't looked at the.err file yet. This instability could even lead to clues why the epot run is problematic.

The MarchNOepot has completed ~51 years continuously.

This reflects ~20years per day which is a n improvement over ESM1.5 which was getting ~16 years per day.

@MartinDix @rml599gh @har917 @tiloz @ccarouge

[JhanSrbinovsky]

I've just restarted MarchEpot as it looks like a ball was dropped in restarted run - there's basically no evidence of it having happened. The PBS log just stops.

OK - so it started and stopped again, same deal. payu is looking for something @blimlim, you know your way around payu better than I do, can you please have a look if anything stands out to you?

[har917]

@JhanSrbinovsky Assuming that I've got the offset to Pearse's long run correct then:

• there's no discernible ocean cooling in the new runs (perhaps a modest drop during the first 10 years or so)
• both are very similar to Pearse's long run at the same point in model time (so almost within internal variability)
• MarchEpot and MarchNOEpot are identical in the ocean variables (and in river flow into ocean)

I'm not sure whether being identical implies that the Epot change is irrelevant or that we've (again) not picked up the appropriate UM7 code and/or executable.

EDIT: Also to confirm that the atmospheric output - as evidenced by TOA energy balance, total cloud, surface temperatures and precip - is also identical over the first 9 years. (MarchEpot failed conversion in year 011, MarchNOEpot in year 009)

[blimlim]

Hi @JhanSrbinovsky. I've taken a look at the control directory for the MarchEpot run.
The latest restart directory /scratch/p66/jxs599/access-esm/archive/MarchEpot-preindustrial-sapphire-rapids-9d8695c2/restart033 is incomplete: it's missing ocean and ice restarts, causing the model to fail when it starts.

To get around this, can you try deleting restart033. On rerunning, the model will pick up the previous restart032, and then hopefully will be able to run smoothly from there.

For background, this is likely caused by a CICE issue. When it crashes, it erroneously reports error code 0, causing payu to view the run as being successful. Payu then puts together an incomplete restart directory, leading to the issue you've encountered. It's on my todo list to port over the fixes added to CM2's CICE in order to avoid this.

[JhanSrbinovsky]

Thanks @blimlim - I was hoping restarting from an earlieryer was this easy.

@har917 - I'll plug the gaps in the converted files. Looks to me like like you have the offset correct.

MarchEpot and MarchNOEpot are identical in the ocean variables (and in river flow into ocean)

But you've just shown that the mean temp. is different? How can everything else be the same?

TOA eval etc identical for first 9 years and then diverges? Both things are crazy.

@blimlim if its possible possible can you please point us to where you can see the code built by pr43-17 and pr43-18? Even just ^/src/atmosphere/CABLE/src/science/canopy/cable_canopy.F90 of each

[JhanSrbinovsky]

@har917 (MarchEpot failed conversion in year 011

MarchNOEpot in year 009 crazy errroabout Iris wanting micro=second precision

back to back use of script esm1p5_convert_nc on back to back simulations

[har917]

@JhanSrbinovsky For clarity - up to the point that I can easily assess (i.e. up to year 9 after the start) MarchEpot and MarchEpot are identical. The difference seen the plot above is from Pearse's long run (blue is Pearse's run, red and green are the two new runs and overlay each other - to numerical precision in the output they are exactly equal in all vars that I've looked at).

I haven't looked beyond year 9 as we hit failed conversion issues in the atmospheric output - the ocean output that I'm using doesn't suffer from the issue so I can look at longer periods. It will be interesting to see how the Epot and NOEpot runs behave once you successfully restart - I would not be surprised if they diverged.

[JhanSrbinovsky]

@har917 - even for first 9 years it sounds like something has gone wrong!

[har917]

@JhanSrbinovsky I was expecting the MarchEpot and MarchNOEpot runs to be different but very similar - so yes being identical indicates that something wasn't correct in the codebase as used and/or the build or that epot is actually irrelevant in the coupled model.

[JhanSrbinovsky]

@har917 I cant answer. The only place I have control over this is in the ACCESS-ESM1.6 PR spack.yaml and specifying the commits of the two versions of the code. And I've just checked this. The only way this would make sense from what we "know" is if we are riding on the lower limit all of the time

[JhanSrbinovsky]

@har917 I'll look into writing out canopy%epot to see if that gives us any clues

[JhanSrbinovsky]

@har917 - it looks like wetfac_cs is always hitting limit of 1. epot is showing some variation. I'll have to look into it more when I get back