Missing variables in rerun file *veg.nc

[YuchenSun33]

Hi

I am trying to branch off a simulation from an previous AWIESM-2 gamma PI experiment.
The error message is:

0: Reading tslclim from unit 92
0: IO_read_stream: failed to read from rerun file: burned_YC_acid_ag_wood
0: IO_read_stream: failed to read from rerun file: burned_YC_water_ag_green
0: IO_read_stream: failed to read from rerun file: burned_YC_water_ag_wood
0: IO_read_stream: failed to read from rerun file: burned_YC_ethanol_ag_green
0: IO_read_stream: failed to read from rerun file: burned_YC_ethanol_ag_wood
0: IO_read_stream: failed to read from rerun file: box_burned_frac_diag_avg
0: IO_read_stream: failed to read from rerun file: dist_nitrogen_2_atmos
0: IO_read_stream: failed to read from rerun file: dist_nitrogen_2_sminn
0: IO_read_stream: failed to read from rerun file: burned_YC_acid_ag_green
0: IO_read_stream: failed to read from rerun file: frac_1hr_wood
0: IO_read_stream: failed to read from rerun file: frac_10hr_wood
0: IO_read_stream: failed to read from rerun file: frac_100hr_wood
0: IO_read_stream: failed to read from rerun file: frac_1000hr_wood
0: IO_read_stream: failed to read from rerun file: NI_acc
0: IO_read_stream: failed to read from rerun file: burned_YC_nonsoluble_ag_green
0: IO_read_stream: failed to read from rerun file: burned_YC_nonsoluble_ag_wood
0: IO_read_stream: failed to read from rerun file: burned_Cpool_litter_wood_ag
0: IO_read_stream: failed to read from rerun file: burned_Cpool_litter_green_ag
0: IO_read_stream: failed to read from rerun file: burned_Cpool_green
0: IO_read_stream: failed to read from rerun file: postfire_mortality
0: IO_read_stream: failed to read from rerun file: Box_flux_harvest_2atmos
0:
0: ================================================================================
0:
0: FATAL ERROR in IO_read_stream: variable expected in rerun file 'restart_PI_CTRL_wiso_veg.nc' is not present: Box_flux_harvest_2atmos
0: FINISH called from PE: 0

It seems like some variables are missing in the rerun file. Is it possible to fix this?
Machine: Levante
Model: awiesm-2.1-wiso

[mandresm]

Can you provide the runscript, and a path to the experiment?

[mandresm]

@pgierz, can you help with this?

[pgierz]

Hi @YuchenSun33, we need some paths to experiments (and branch offs) to take a better look at this. It looks from the missing file error that you switched JSBACH modes. That is just a guess, though.

[YuchenSun33]

Thank you for getting back to me.
My EXP path is: /work/ba0989/a270107/esm_tools/awiesm-2.1-wiso/EXP/PI_CTRL_wiso
The restart files are from: /home/a/a270064/bb1029/AWIESM2.1-EXP/pi-50/restart

[YuchenSun33]

Hi @pgierz, this problem was solved by using the namelist.jsbach from pi-50. Now I met a new problem which is missing some rerun files from echam and jsbach. Here is the warning message from esm_tools:

WARNING: File not found: /home/a/a270064/bb1029/AWIESM2.1-EXP/pi-50/restart/echam/restart_pi-50_rad_23401231.nc
WARNING: File not found: /home/a/a270064/bb1029/AWIESM2.1-EXP/pi-50/restart/jsbach/restart_pi-50_23401231235230_nitro.nc
WARNING: File not found: /home/a/a270064/bb1029/AWIESM2.1-EXP/pi-50/restart/jsbach/restart_pi-50_23401231235230_land.nc
WARNING: File not found: /home/a/a270064/bb1029/AWIESM2.1-EXP/pi-50/restart/jsbach/restart_pi-50_23401231235230_la_wiso.nc
WARNING: File not found: /home/a/a270064/bb1029/AWIESM2.1-EXP/pi-50/restart/jsbach/restart_pi-50_23401231235230_sf_wiso.nc

[mandresm]

Is it running or is it just warning?

[YuchenSun33]

No. The experiment can’t start because of missing restart*sf_wiso.nc

[YuchenSun33]

Now I tried to set lwiso_rerun: false in the yaml file, which can avoid showing these "File not found" messages. But the experiment still couldn't run. I dig a little bit in the log file and found these lines that might be related:
618: reading restart PARALLEL for ssh at /work/ba0989/a270107/esm_tools/awiesm-2.1-wiso/EXP/PI_CTRL_wiso_2/run_20010101-20011231/work/fesom.2340.oce.restart/ssh.nc
618: error in line 261 io_netcdf_file_module.F90 No such file or directory
Would you happen to have any idea about this?
My EXP path is: /work/ba0989/a270107/esm_tools/awiesm-2.1-wiso/EXP/PI_CTRL_wiso_2

[nwieters]

Hi @YuchenSun33, @mandresm, @pgierz,

could it be that the experiment from which you want to restart from have been done with a different fesom version?

If I have a look into the fesom folder of your awiesm model_dir (/work/ba0989/a270107/esm_tools/awiesm-2.1-wiso/fesom-2.1/) it says it is using fesom from the branch wiso-fesom2.5. I think this version expects the restart files in its own Netcdf files (ssh.nc).

By default, awiesm-2.1-wiso will install fesom-2.1 from the branch wiso that does not have a Fortran file io_netcdf_file_module.F90.

This is maybe related to changes you have done in fesom-2.1 yaml file?
/work/ba0989/a270107/esm_tools/configs/components/fesom/fesom-2.1.yaml

When I do git status in your esm-tools installation it shows changes in this file
(components/fesom/fesom-2.1.yaml).
The changes there also include the fesom branch.

[YuchenSun33]

Deal all,
Thank you very much for all of your kind help. I have now fixed the problems.
The FESOM version I used is wiso-fesom2.5. When restart from a 2.1-wiso experiment, the needed rerun files are different.
From the ECHAM side, I have to set lwiso_rerun to false for the initial year and set back to true for the second year manually (This will not automatically modified by esmtools).
From the FESOM side, I separated the variables in the original fesom 2.1-wiso restart files and rearranged their dimensions. This ensures that fesom2.5 can access the necessary files without issues.

[pgierz]

Hi @YuchenSun33,

OK, glad that it is working. However, I would generally strongly recommend against changing versions of one part of your model in the middle of an experiment. Particularly with FESOM. There are enough things that can be nasty and bite you in the butt ;-)

For the ECHAM side, have you been able to do any branch off of an old run to a new run while using wiso? That might be something to look into, that should be possible, and if it is, we should probably document that nicely.

[YuchenSun33]

Hi @pgierz ,

In the beginning, I am not able to branch off of an old run (awiesm2 gamma) to a new run while using wiso, because one file restart_*_sf_wiso.nc is missing.
I can find this restart file In the unknown folder, but not for the latest year. In my case, I want to continue the run from the end of year 2340 of /home/a/a270064/bb1029/AWIESM2.1-EXP/pi-50. The file restart_pi-50_23401231235230_sf_wiso.nc is missing in restart/jsbach. There is a file named restart_pi-50_sf_wiso.nc.bak_23400101-23401231 in the unkown folder, however, it is not the restart file for the end of the year 2340, but the year 2339.
I just found the correct file at run_23400101-23401231/work/restart_pi-50_sf_wiso.nc. And now the model is running smoothly with wiso.

[pgierz]

@nwieters: if you want to practice with the old syntax before we at some point implement the file dict way, this is a very nice example of a file not being defined in the right place, and instead being dumped into unknown