Merge branch 'main' into add-more-datasets-to-recipe-easy-ipcc

Author: jlenh

.zenodo.json

@@ -66,6 +66,11 @@
       "affiliation": "University of Bremen, Germany",
       "name": "Adeniyi, Kemisola"
     },
+    {
+      "affiliation": "University of Bremen, Germany",
+      "name": "Castellani, Giulia",
+      "orcid": "0000-0001-6151-015X"
+    },
     {
       "affiliation": "ISAC-CNR, Italy",
       "name": "Arnone, Enrico",
@@ -366,6 +371,10 @@
       "affiliation": "DLR, Germany",
       "name": "Sarauer, Ellen"
     },
+    {
+      "affiliation": "University of Bremen, Germany",
+      "name": "Schulze, Kirsten"
+    },
     {
       "affiliation": "University of Reading, UK",
       "name": "Roberts, Charles",

CITATION.cff

@@ -174,6 +174,10 @@ authors:
     affiliation: "University of Bremen, Germany"
     family-names: Gier
     given-names: Bettina
+  -
+    affiliation: "University of Bremen, Germany"
+    family-names: Castellani
+    given-names: Giulia
   -
     affiliation: "Met Office, UK"
     family-names: Gillett
@@ -358,6 +362,10 @@ authors:
     family-names: Weigel
     given-names: Katja
     orcid: "https://orcid.org/0000-0001-6133-7801"
+  -
+    affiliation: "University of Bremen, Germany"
+    family-names: Schulze
+    given-names: Kirsten
   -
     affiliation: "DLR, Germany"
     family-names: Sarauer

conda-linux-64.lock

@@ -134,3 +134,185 @@ high-level interface to the `Seaborn <https://seaborn.pydata.org>`__ package
 which can also be used to create a large variety of different plots.
 
 See also :ref:`general_purpose_diags`.
+
+Debugging FAQ
+=============
+
+Something's going wrong in preprocessing, how can I find out where?
+-------------------------------------------------------------------
+
+Sometimes there is a problem in the preprocessing, often because of faulty data.
+
+To locate the problem, the first step is to edit your `config-user.yml` to set
+
+.. code-block:: yaml
+
+   log_level: debug
+
+Then run `esmvaltool` again and try to understand the error message. Don't
+forget that you can open the error log in the run directory instead of working
+only with the terminal output.  Usually, the error log ends in a python
+traceback.  See for example `here <https://realpython.com/python-traceback/>`_
+for an introduction to understanding tracebacks.
+
+It can also be helpful to look in the error log directly above the traceback for
+more information on what caused the problem.
+
+If the error message is not enough to diagnose the problem, you can "spread out"
+the effects of the different preprocessors. For that, edit again
+`config-user.yml` to set
+
+.. code-block:: yaml
+
+   save_intermediary_cubes: true
+   remove_preproc_dir: false
+
+This way, you can inspect the output of every preprocessing step individually by
+looking at the individual outputs in the `preproc` directory. Usually, the last
+existing file can give you a clue on what went wrong.
+
+.. warning::
+
+   This can be costly both in storage space and computing time, so be careful to
+   use as small a dataset as possible to diagnose your problem and to deactivate
+   the setting when you are done debugging!
+
+I get a `iris.exceptions.ConcatenateError`. What's the problem?
+---------------------------------------------------------------
+
+Broadly speaking, the problem is that data that was spread over different files
+was supposed to be part of one dataset, but doesn't follow the same conventions.
+This can be due to different coordinates, different attributes, or a number of
+other differences. Unfortunately, it can be difficult to understand what exactly
+is the difference between the data.
+
+To diagnose the problem, first, figure out which netcdf files cause the
+issue. If it happens in the preprocessing, refer to `this FAQ
+<somethings-going-wrong-in-preprocessing-how-can-I-find-out-where>`_ for some
+pointers.
+
+Next, try to understand the differences between the two files. Perhaps the error
+message already pointed to the area of difference, for example some attributes
+or particular coordinates. In that case you can focus on those first.
+
+How can I compare two (or more) netcdf files?
+---------------------------------------------
+
+There are three popular tools/methods to understand the differences between
+netcdf files. The first two, `looking at the metadata of the individual files
+<inspecting-the-netcdf-file-headers>`_ and `visualizing their data
+<visualizing-the-data>`_, are almost always possible and can give you a quick
+idea of what's going on. They are, however, often not suited to detailed
+comparisons, for example when the actual data contains subtle problems or when
+the problem is in the values of coordinates. In those cases, turn to `a more
+in-depth look <comparing-data-with-iris>`_ at the data. This last method also
+works better when you have a larger collection of files and are not sure where
+exactly the problem lies.
+
+Inspecting the netcdf file headers
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can use the `ncdump` utility to look at the metadata of netcdf files. More
+specifically, `ncdump -h my_file.nc` will output the so-called header of the
+file that contains information about all the variables present in the file,
+along with their units, etc. This includes the coordinate variables. It also
+will give you all the global, ie file level, and local, ie variable level
+attributes.  It will, however, not give you the coordinate values or the data
+values itself.  It will also not help you with comparison, so you can only
+compare the output for two files manually.
+
+Nevertheless, when you want to get a quick overview, or when you have some idea
+what to look for, or when the other methods don't work, this is a good starting
+point.
+
+Visualizing the data
+~~~~~~~~~~~~~~~~~~~~
+
+Often a quick look at the data can reveal the most glaring problems, like
+missing or wrong masks, or wrong units.  There are a number of tools that can be
+used for this, but one of the most basic and almost universally available ones
+is `ncview`. This can also be installed with conda from conda-forge.
+
+Just do a quick `ncview my_file.nc` to open the gui. Then select the variable
+and use the navigation buttons to flip through the timesteps and, in the case of
+3d data, the levels.
+
+Again, this tool doesn't help you with comparison, but you can open two
+instances to compare two files in side-by-side.
+
+Comparing data with iris
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+`Iris <https://scitools.org.uk/iris/docs/latest/>`_ is a powerful python
+framework for working with `CF <http://cfconventions.org/>`_ compliant netcdf
+files. It also includes some support for other formats like grib, and pp files.
+Iris is the foundation for esmvaltool, but it can also be used on its own. Two
+convenient ways to do that are `ipython <https://ipython.org/>`_ and `jupyter
+notebooks <https://jupyter.org/>`_. This FAQ cannot cover any of these tools in
+great detail, so please refer to their respective documentation for an overview
+and in-depth questions. Instead, here we will give just a quick way to
+understand differences in a set of netcdf files with iris.
+
+Iris loads data with one of several functions whose names start with
+`load`. These functions take a single string or a list of strings that are
+interpreted as filenames. They may also contain directories and globs, like
+`./my_data/tas_*.nc` to load all surface temperature data from the `my_data`
+subdirectory of the current working directory. The iris function will return
+either a single `Cube
+<https://scitools.org.uk/iris/docs/latest/iris/iris/cube.html#iris.cube.Cube>`_,
+or a list of cubes in the form of a `CubeList
+<https://scitools.org.uk/iris/docs/latest/iris/iris/cube.html#iris.cube.CubeList>`_.
+
+Often, we want to combine different parts of dataset to form the whole dataset
+again. This is accomplished in iris using either a `merge` or a
+`concatenate`. Refer to `the iris user guide
+<https://scitools.org.uk/iris/docs/latest/userguide/merge_and_concat.html>`_ to
+understand the difference.
+
+In fact, it is this `concatenate` that has failed you when you are reading this
+FAQ entry.
+
+To find out what went wrong, let's assume we have boiled down the problem to the
+two files `tas_1969.nc` and `tas_1970.nc`. Then one way to learn more about the
+issue is the following.
+
+.. code-block:: python
+
+   import iris
+   cube_list = iris.load('tas_19*.nc')
+   cube_list.concatenate_cube()
+
+This will likely produce the exact same error that we started with. While we do
+want to do a concatenation, since both files contain several timesteps each, the
+output of the corresponding merge function is often more illuminating. So let's
+give that a try.
+
+.. code-block:: python
+
+   cube_list.merge_cube()
+
+Hopefully, that gives you a better idea of the problem.
+
+If you are dealing with more than two files it can be difficult to figure out
+which files are actually the culprits. In that case try
+
+.. code-block:: python
+
+   import iris
+   cube_list = iris.load('tas_*.nc')
+   concatenated_cube_list = cube_list.concatenate()
+   print(concatenated_cube_list)
+
+This will concatenate all the consecutive bits it can manage and keep only those
+parts separate, that cannot be merged. This way, you can often boil down the
+problem to only two or a few cubes. Then you can diagnose those in the same way
+as discussed above, eg
+
+.. code-block:: python
+
+   concatenated_cube_list.concatenate_cube()  # or
+   concatenated_cube_list.merge_cube()
+
+You can also inspect the attributes and coordinates of the resulting cubes with
+a particular focus on the areas of problems pointed to by the output of
+`merge_cube`.

doc/sphinx/source/faq.rst

@@ -311,8 +311,10 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | ESACCI-AEROSOL               | abs550aer, od550aer, od550aerStderr, od550lt1aer, od870aer, od870aerStderr (aero)                    |   2  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
+| ESACCI-BIOMASS               | agb (Lyr, frequency=yr)                                                                              |   2  | Python          |
++------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | ESACCI-CLOUD                 | clivi, clt, cltStderr, clwvi, lwp, rlut, rlutcs, rsut, rsutcs, rsdt, rlus, rsus, rsuscs (Amon),      |   2  | Python          |
-|                              | clt, clwvi, cod (day)                                                                                |   2  |                 |
+|                              | clt, clwvi, cod (day)                                                                                |      |                 |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | ESACCI-FIRE                  | burntArea (Lmon)                                                                                     |   2  | NCL             |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
@@ -450,6 +452,8 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | NOAA-GML-SURFACE-FLASK-CO2   | co2s (Amon)                                                                                          |   2  | Python          |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
+| NOAA-GML-SURFACE-FLASK-N2O   | n2os (Amon)                                                                                          |   2  | Python          |
++------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | NOAAGlobalTemp               | tasa (Amon)                                                                                          |   2  | Python          |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 | NSIDC-0116-[nh|sh] [#note4]_ | usi, vsi (day)                                                                                       |   3  | Python          |
@@ -489,6 +493,9 @@ A list of the datasets for which a CMORizers is available is provided in the fol
 | WOA                          | thetao, so, tos, sos (Omon)                                                                          |   2  | Python          |
 |                              | no3, o2, po4, si (Oyr)                                                                               |      |                 |
 +------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
+| Yang2020                     | dpn2o, no2flux (Omon)                                                                                |   2  | Python          |
+|                              | areacella (fx)                                                                                       |      |                 |
++------------------------------+------------------------------------------------------------------------------------------------------+------+-----------------+
 
 .. [#t3] We obtained permission from the dataset provider to share this dataset
          among ESMValTool users on HPC systems.