Error when running analysis on ACCESS-ESM1-5 data

[jumin94]

Hello! I am not 100% new to ESMValTool, but I think that I was just lucky with my first recipes and didn't encounter the problems I am encountering now.

I had previously run a recipe using ACCESS-ESM1-5 data from CMIP6 with no problems. Now, I find the problem that you can see below. I am wondering if there's a general way to address these issues (so that I don't have to start a discussion every time I have an error like this)

>`2024-08-06 13:35:57,161 UTC [129573] ERROR   Failed to run preprocessor function 'add_supplementary_variables' on the data
[<iris 'Cube' of air_temperature / (K) (time: 1800; air_pressure: 19; latitude: 144; longitude: 192)>]
loaded from original input file(s)
[LocalFile('/climca/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/Amon/ta/gn/v20191115/ta_Amon_ACCESS-ESM1-5_historical_r1i1p1f1_gn_195001-201412.nc'),
 LocalFile('/climca/data/CMIP6/ScenarioMIP/CSIRO/ACCESS-ESM1-5/ssp585/r1i1p1f1/Amon/ta/gn/v20210318/ta_Amon_ACCESS-ESM1-5_ssp585_r1i1p1f1_gn_201501-210012.nc'),
 LocalFile('/climca/data/CMIP6/CMIP/CSIRO/ACCESS-ESM1-5/historical/r1i1p1f1/fx/areacella/gn/v20191115/areacella_fx_ACCESS-ESM1-5_historical_r1i1p1f1_gn.nc')]
with function argument(s)
supplementary_cubes = [<iris 'Cube' of cell_area / (m2) (latitude: 145; longitude: 192)>]
2024-08-06 13:35:57,897 UTC [129378] INFO    Maximum memory used (estimate): 1.1 GB
2024-08-06 13:35:57,899 UTC [129378] INFO    Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur.
2024-08-06 13:35:57,901 UTC [129378] ERROR   Program terminated abnormally, see stack trace below for more information:
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_task.py", line 816, in _run_task
    output_files = task.run()
                   ^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_task.py", line 264, in run
    self.output_files = self._run(input_files)
                        ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 661, in _run
    product.apply(step, self.debug)
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 469, in apply
    self.cubes = preprocess(self.cubes, step,
                            ^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 479, in cubes
    self._cubes = [ds.load() for ds in self.datasets]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 479, in <listcomp>
    self._cubes = [ds.load() for ds in self.datasets]
                   ^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/dataset.py", line 692, in load
    cubes = preprocess(
            ^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 378, in preprocess
    result.append(_run_preproc_function(function, item, settings,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/preprocessor/__init__.py", line 323, in _run_preproc_function
    return function(items, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/preprocessor/_supplementary_vars.py", line 134, in add_supplementary_variables
    add_cell_measure(cube, supplementary_cube, measure_name)
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/preprocessor/_supplementary_vars.py", line 76, in add_cell_measure
    cube.add_cell_measure(measure, range(start_dim, cube.ndim))
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/iris/cube.py", line 1640, in add_cell_measure
    data_dims = self._check_multi_dim_metadata(cell_measure, data_dims)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/iris/cube.py", line 1523, in _check_multi_dim_metadata
    raise iris.exceptions.CannotAddError(
iris.exceptions.CannotAddError: Unequal lengths. Cube dimension 2 => 144; metadata 'cell_area' dimension 0 => 145.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_main.py", line 518, in run
    fire.Fire(ESMValTool())
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_main.py", line 405, in run
    self._run(recipe, session)
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_main.py", line 447, in _run
    process_recipe(recipe_file=recipe, session=session)
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_main.py", line 130, in process_recipe
    recipe.run()
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_recipe/recipe.py", line 1075, in run
    self.tasks.run(max_parallel_tasks=self.session['max_parallel_tasks'])
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_task.py", line 738, in run
    self._run_parallel(address, max_parallel_tasks)
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_task.py", line 782, in _run_parallel
    _copy_results(task, running[task])
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/site-packages/esmvalcore/_task.py", line 805, in _copy_results
    task.output_files, task.products = future.get()
                                       ^^^^^^^^^^^^
  File "/opt/miniforge3/envs/esmvaltool/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
iris.exceptions.CannotAddError: Unequal lengths. Cube dimension 2 => 144; metadata 'cell_area' dimension 0 => 145.
2024-08-06 13:35:57,907 UTC [129378] INFO    `

[bouweandela]

@rbeucher Would you or one of your team be able to help out here? It looks like the areacella file is on a different grid from the ta file.

[flicj191]

I'll have a look once our systems is up and accessible

[valeriupredoi]

just for the record (if it pans out to be an iris issue), could you pls tell us what version of iris you using via conda list iris please @jumin94 🍺

[flicj191]

Hello! sorry I got distracted, I have checked the files I can see and it does look like that grid is different, (latitude dimension is 145 in area while 144 in the variable), I have checked some other variables for that model and some are 144 some are 145. I'll post to the ACCESS community to look for answers, and share here. But in short it does look like a data issue. Cheers @jumin94 !

[rbeucher]

Thanks @flicj191. Good catch

[flicj191]

Some fields in the atmosphere is offset by half a grid spacing and should have another cell measure see post. CMIP6 though only has one area file