Merge branch 'main' into python3.14

Author: valeriupredoi

CITATION.cff

@@ -455,11 +455,11 @@ authors:
 
 
 cff-version: 1.2.0
-date-released: 2025-10-20
+date-released: 2026-03-13
 doi: "10.5281/zenodo.3401363"
 license: "Apache-2.0"
 message: "If you use this software, please cite it using these metadata."
 repository-code: "https://github.com/ESMValGroup/ESMValTool/"
 title: ESMValTool
-version: "v2.13.0"
+version: "v2.14.0"
 ...

doc/sphinx/source/changelog.rst

@@ -3,6 +3,138 @@
 Changelog
 =========
 
+.. _changelog-v2-14-0:
+
+v2.14.0
+-------
+
+Highlights
+~~~~~~~~~~
+
+- New diagnostics/recipes:
+   - Calculation of Lamb weathertypes (:pull:`4231`)
+   - Histogram plots (:pull:`4329`)
+   - Hour of maximum precipitation (:pull:`4334`)
+- Support for new observational datasets:
+   - ESACCI SNOW (:pull:`3542`)
+   - Improved support for ESACCI AEROSOL (:pull:`3629`)
+   - EN4 (:pull:`4193`)
+- We modernized our development set-up and are now using `Ruff <https://docs.astral.sh/ruff/>`__ for :ref:`code_quality` checks and formatting.
+
+This release includes
+
+Backwards incompatible changes
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  Update the CMORizer command (:pull:`4317`, :pull:`4326`) by :user:`bouweandela`
+
+   .. admonition:: Upgrade instructions
+
+      The ``--config-file`` argument to the ``esmvaltool data`` commands has
+      been removed. Please use these commands with the ``--config-dir``
+      argument to specify the output directory, custom CMOR tables, or Dask
+      configuration, and with the ``--original-data-dir`` argument to point the
+      tool to the directory where downloaded original data should be saved.
+
+Bug fixes
+~~~~~~~~~
+
+-  Fix recipe_enso_climatology1 - remove collect diagnostic (:pull:`4221`) by :user:`flicj191`
+-  Rename ``modeling_realm`` to ``type`` in CMORizers (:pull:`4319`) by :user:`bouweandela`
+-  Fix ``multi_datasets.py`` so it can handle different projections (:pull:`4358`) by :user:`schlunma`
+
+Broken or retired recipes
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  Remove ``recipe_russel18jgr.yml`` from list of broken recipes (:pull:`4351`) by :user:`schlunma`
+-  Retire ``recipe_climwip_brunner2019_med.yml`` (:pull:`4350`) by :user:`schlunma`
+-  Add ``recipe_kcs.yml`` to list of broken recipes (:pull:`4354`) by :user:`schlunma`
+-  Retire Wenzel recipes (:pull:`4357`) by :user:`schlunma`
+-  Remove ``recipe_check_obs.yml`` from list of broken recipes (:pull:`4376`) by :user:`schlunma`
+-  Retire ``recipe_eyring*jgr.yml`` (:pull:`4379`) by :user:`schlunma`
+
+Documentation
+~~~~~~~~~~~~~
+
+-  Separate ESMValTool and ESMValCore documentation (:pull:`3914`) by :user:`flicj191`
+-  Fix PDF documentation build (:pull:`4236`) by :user:`bouweandela`
+-  Update links to ESMValCore configuration (:pull:`4249`) by :user:`bouweandela`
+-  Document that we follow SPEC 0 (:pull:`4246`) by :user:`bouweandela`
+-  Fix stable documentation build (:pull:`4253`) by :user:`bouweandela`
+-  Update some easy to fix links in documentation (:pull:`4256`) by :user:`katjaweigel`
+-  Add link checker github action for documentation (:pull:`4251`) by :user:`flicj191`
+-  Add ENES-RI Zenodo community (:pull:`4254`) by :user:`bouweandela`
+-  Schedule for link checker and update for recent report (:pull:`4305`) by :user:`flicj191`
+-  Pin sphinx<9 (:pull:`4315`) by :user:`valeriupredoi`
+-  Add Matomo page view tracking for the documentation (:pull:`4320`) by :user:`bouweandela`
+-  Fix matomo integration (:pull:`4322`) by :user:`bouweandela`
+-  Remove mentions of the deprecated config-developer.yml file from the documentation (:pull:`4335`) by :user:`bouweandela`
+-  Documentation update -remove gensidebar (:pull:`4345`) by :user:`flicj191`
+-  Update documentation on defining custom variables (:pull:`4349`) by :user:`bouweandela`
+-  Update contact info (:pull:`4348`) by :user:`axel-lauer`
+-  Remove mention of deprecated ``esmvaltool config get_config_user`` command (:pull:`4366`) by :user:`schlunma`
+-  Update contact email in documentation (:pull:`4383`) by :user:`axel-lauer`
+
+Diagnostics
+~~~~~~~~~~~
+
+-  Calculating Lamb weathertypes for CMIP5, CMIP6, SMILES and EURO-CORDEX (:pull:`4231`) by :user:`thomaskroi1996`
+-  Add diagnostic to plot a histogram (:pull:`4329`) by :user:`LisaBock`
+-  Add diagnostic to calculate hour of maximum precipitation (:pull:`4334`) by :user:`schlunma`
+-  Make emergent constraints diagnostics compatible with Pandas>=3 (:pull:`4356`) by :user:`schlunma`
+-  Allow specifying ``axes_kwargs`` in ``monitor/multi_datasets.py`` (:pull:`4365`) by :user:`schlunma`
+
+Observational and re-analysis dataset support
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+-  ESACCI SNOW CMORizer (:pull:`3542`) by :user:`axel-lauer`
+-  Update CMORizer CERES-EBAF to v4.2 (:pull:`3360`) by :user:`axel-lauer`
+-  Fix JRA-55 downloader (:pull:`4191`) by :user:`LisaBock`
+-  EN4 cmorizer (:pull:`4193`) by :user:`hanellis`
+-  New ESACCI AEROSOL CMORizer (Python version) (:pull:`3629`) by :user:`axel-lauer`
+-  Extension of ESA CCI OZONE CMORizer (:pull:`4125`) by :user:`axel-lauer`
+
+Release
+~~~~~~~
+
+-  Modernize release notes script (:pull:`4336`) by :user:`schlunma`
+
+Automatic testing
+~~~~~~~~~~~~~~~~~
+
+-  Run Github Action install from conda with Python 3.13 also (:pull:`4229`) by :user:`valeriupredoi`
+-  Fix WOA CMORizer tests (:pull:`4243`) by :user:`valeriupredoi`
+-  Update recipe tests to align with ESMValCore development (:pull:`4268`) by :user:`bouweandela`
+-  Fix tests for new esmvalcore config (:pull:`4252`) by :user:`valeriupredoi`
+-  Clean up OSX Github Action tests (:pull:`4272`) by :user:`valeriupredoi`
+-  Update recipe load test so it works with the latest ESMValCore (:pull:`4294`) by :user:`bouweandela`
+-  Increase CirleCI container resource to medium+ (:pull:`4364`) by :user:`valeriupredoi`
+-  Update batch job generation script with experiences from latest tests (:pull:`4378`) by :user:`schlunma`
+
+Installation
+~~~~~~~~~~~~
+
+-  Use ``pip install --no-deps`` to avoid overwiting packages from the conda environment (:pull:`4182`) by :user:`bouweandela`
+-  Drop support for Python 3.11 (:pull:`4273`) by :user:`bouweandela`
+-  Pin iris <3.14.1 to comply with iris-esmf-regrid 0.13 (:pull:`4284`) by :user:`valeriupredoi`
+-  Update package build and deploy (:pull:`4289`) by :user:`bouweandela`
+-  Drop ``ruamel.yaml`` as a dependency (:pull:`4295`) by :user:`bouweandela`
+-  Unpin iris (:pull:`4304`) by :user:`valeriupredoi`
+-  Remove R development dependencies (:pull:`4306`) by :user:`valeriupredoi`
+-  Update environment.yml with R dependency r-r.utils for recipe_martin18grl and recipe_spei (:pull:`4355`) by :user:`katjaweigel`
+-  Pin numpy >=2 (:pull:`4375`) by :user:`valeriupredoi`
+
+Improvements
+~~~~~~~~~~~~
+
+-  Switch from Prospector to ruff (:pull:`4209`) by :user:`bouweandela`
+-  Fix Codacy configuration (:pull:`4262`) by :user:`bouweandela`
+-  Apply automatic fixes for style issues (:pull:`4263`) by :user:`bouweandela`
+-  Update obs4MIPs dataset names (:pull:`4321`, :pull:`4377`) by :user:`bouweandela` and :user:`valeriupredoi`
+-  Add ``keep_group_coordinates: true`` to recipes that fail without it (:pull:`4352`) by :user:`schlunma`
+-  Load configuration before accessing CMOR tables (:pull:`4363`) by :user:`bouweandela`
+-  Avoid failing tests on coverage upload from forks (:pull:`4371`) by :user:`bouweandela`
+
 .. _changelog-v2-13-0:
 
 v2.13.0

esmvaltool/cmorizers/data/cmor_config/ESACCI-OZONE.yml

@@ -5,16 +5,40 @@ attributes:
   tier: 2
   type: sat
   project_id: OBS6
-  source: "https://cds.climate.copernicus.eu/datasets/satellite-ozone-v1"
+  source: "https://cds.climate.copernicus.eu/datasets/satellite-ozone-v1 and https://webdav.aeronomie.be/guest/o3_cci/webdata/Nadir_Profiles/L3/IASI_MG_FORLI/"
   reference: "esacci-ozone"
   comment: ""
 
 variables:
-  toz:
+  # Note: Do not change variable names, these are used to handle the different
+  # datasets in the formatting script esacci_ozone.py.
+  toz_gto_ecv:
+    version: L3-GTO-ECV
     mip: AERmon
+    output: toz
     raw: total_ozone_column
-    filename: C3S-L3_OZONE-O3_PRODUCTS-MERGED_UV-MERGED-ALG-MONTHLY-v2000.nc #The filename needs also "YYYYMM-" as prefix added during the cmorization
-  o3:
+    filename: "{year}{month}-C3S-L3_OZONE-O3_PRODUCTS-MERGED_UV-MERGED-ALG-MONTHLY-v2000.nc"
+  o3_sage_omps:
+    version: L3-SAGE-OMPS
     mip: AERmon
+    output: o3
     raw: merged_ozone_concentration
-    filename: C3S-L3_OZONE-O3_PRODUCTS-CONC_MZM-MERGED-ALG-MONTHLY-v0008.nc #The filename needs also "YYYYMM-" as prefix added during the cmorization
+    filename: "{year}{month}-C3S-L3_OZONE-O3_PRODUCTS-CONC_MZM-MERGED-ALG-MONTHLY-v0008.nc"
+  o3_megridop:
+    version: L3-MEGRIDOP
+    mip: AERmon
+    output: o3
+    raw: merged_ozone_concentration
+    filename: "{year}{month}-C3S-L3_OZONE-O3_PRODUCTS-CONC_LLG-MERGED-ALG-MONTHLY-v0004.nc"
+  o3_iasi:
+    version: L3-IASI
+    mip: AERmon
+    output: o3
+    raw: O3_partial_column_profile
+    filename: "IASI_FORLI_O3_MERGED_{year}{month}*_V1.0.nc"
+  toz_iasi:
+    version: L3-IASI
+    mip: AERmon
+    output: toz
+    raw: O3_total_column
+    filename: "IASI_FORLI_O3_MERGED_{year}{month}*_V1.0.nc"

esmvaltool/cmorizers/data/datasets.yml

@@ -573,34 +573,54 @@ datasets:
   ESACCI-OZONE:
     tier: 2
     source: https://cds.climate.copernicus.eu/datasets/satellite-ozone-v1
-    last_access: 2025-02-11
+    last_access: 2026-02-12
     info: |
       Download the data from:
             GTO-ECV total column (variable toz)
             Select the following from the CDS:
             https://cds.climate.copernicus.eu/datasets/satellite-ozone-v1?tab=download
+            Put all files under a single directory (no subdirectories with years).
 
             Processing Level = "Level 3"
             Variable = "Atm. mole content of ozone"
-            Vertical aggregation = "Total column"
-            Sensor = "MERGED-UV"
+            Vertical aggregation = "Total columns UV"
+            Sensor = "GTO-ECV"
             Year = select all (1995-2023)
             Month = select all (1-12)
             Version = "v2000"
 
             SAGE-CCI-OMPS (variable o3)
             Select the following options from the same link:
             https://cds.climate.copernicus.eu/datasets/satellite-ozone-v1?tab=download
+            Put all files under a single directory (no subdirectories with years).
 
             Processing Level = "Level 3"
             Variable = "Mole concentration of ozone in air"
-            Vertical aggregation = " Vertical profiles from limb sensors"
-            Sensor = " CMZM (Monthly zonal mean merged concentration product from limb
-            sensors ACE, GOMOS, MIPAS, OMPS, OSIRIS, SAGE-2 and SCIAMACHY)"
+            Vertical aggregation = "Vertical profiles from limb sensors"
+            Sensor = "SAGE-CCI-OMPS_CONC"
             Year = select all (1984-2022)
             Month = select all (1-12)
             Version = "v0008"
-      Put all files under a single directory (no subdirectories with years).
+
+            MEGRIDOP (variable o3)
+            Select the following options from the same link:
+            https://cds.climate.copernicus.eu/datasets/satellite-ozone-v1?tab=download
+            Put all files under a single directory (no subdirectories with years).
+
+            Processing Level = "Level 3"
+            Variable = "Mole concentration of ozone in air"
+            Vertical aggregation = "Vertical profiles from limb sensors"
+            Sensor = "MEGRIDOP_CONC"
+            Year = select all (2001-2022)
+            Month = select all (1-12)
+            Version = "v0004"
+
+            IASI (variables o3, toz)
+            Download from BIRA WebDAV server: https://webdav.aeronomie.be
+            Path: /guest/o3_cci/webdata/Nadir_Profiles/L3/IASI_MG_FORLI/<yyyy>
+            Username: o3_cci_public
+            No password (leave empty)
+            Download each year <yyyy> into the same folder
 
   ESACCI-SEAICE:
     tier: 2

esmvaltool/cmorizers/data/downloaders/datasets/esacci_ozone.py

@@ -1,11 +1,15 @@
-"""Script to download ESACCI-OZONE from the CDS."""
+"""Script to download ESACCI-OZONE from the CDS and BIRA WebDAV."""
 
 import gzip
 import logging
 import shutil
 import zipfile
+from datetime import datetime
 
 import cdsapi
+from dateutil import relativedelta
+
+from esmvaltool.cmorizers.data.downloaders.wget import WGetDownloader
 
 logger = logging.getLogger(__name__)
 
@@ -26,28 +30,61 @@ def download_dataset(
       the ECMWF account needs to be saved in user's ${HOME} directory.
     - All the files will be saved in Tier2/ESACCI-OZONE.
     """
-    cds_url = "https://cds.climate.copernicus.eu/api"
-
     if dataset == "ESACCI-OZONE":
+        output_folder = (
+            original_data_dir / f"Tier{dataset_info['tier']}" / dataset
+        )
+        output_folder.mkdir(parents=True, exist_ok=True)
+
+        cds_url = "https://cds.climate.copernicus.eu/api"
+
+        if start_date is None:
+            gto_year1 = 1995
+            omps_year1 = 1984
+            megridop_year1 = 2001
+        else:
+            gto_year1 = start_date.year
+            omps_year1 = start_date.year
+            megridop_year1 = start_date.year
+        if end_date is None:
+            gto_year2 = 2024
+            omps_year2 = 2023
+            megridop_year2 = 2023
+        else:
+            gto_year2 = end_date.year
+            omps_year2 = end_date.year
+            megridop_year2 = end_date.year
+
         requests = {
-            "toz": {
+            "toz_gto_ecv": {
                 "processing_level": "level_3",
                 "variable": "atmosphere_mole_content_of_ozone",
-                "vertical_aggregation": "total_column",
-                "sensor": ["merged_uv"],
-                "year": [str(y) for y in range(1995, 2024)],
+                "vertical_aggregation": "total_columns_uv",
+                "sensor": ["gto_ecv"],
+                "year": [str(y) for y in range(gto_year1, gto_year2)],
                 "month": [f"{m:02d}" for m in range(1, 13)],
                 "version": ["v2000"],
             },
-            "o3": {
+            "o3_sage_omps": {
                 "processing_level": "level_3",
                 "variable": "mole_concentration_of_ozone_in_air",
                 "vertical_aggregation": "vertical_profiles_from_limb_sensors",
-                "sensor": ["cmzm"],
-                "year": [str(y) for y in range(1984, 2023)],
+                "sensor": ["sage_cci_omps_conc"],
+                "year": [str(y) for y in range(omps_year1, omps_year2)],
                 "month": [f"{m:02d}" for m in range(1, 13)],
                 "version": ["v0008"],
             },
+            "o3_sage_megridop": {
+                "processing_level": "level_3",
+                "variable": "mole_concentration_of_ozone_in_air",
+                "vertical_aggregation": "vertical_profiles_from_limb_sensors",
+                "sensor": ["megridop_conc"],
+                "year": [
+                    str(y) for y in range(megridop_year1, megridop_year2)
+                ],
+                "month": [f"{m:02d}" for m in range(1, 13)],
+                "version": ["v0004"],
+            },
         }
 
         client = cdsapi.Client(cds_url)
@@ -88,5 +125,41 @@ def download_dataset(
                     with open(output_folder / file_path.stem, "wb") as f_out:
                         shutil.copyfileobj(f_in, f_out)
 
+        # download IASI data from BIRA WebDAV (IASI data not available on CDS)
+        # all the files will be saved by in ${RAWOBS}/Tier2/ESACCI-OZONE
+
+        if start_date is None:
+            start_date = datetime(2008, 1, 1)
+        if end_date is None:
+            end_date = datetime(2023, 12, 31)
+
+        downloader = WGetDownloader(
+            original_data_dir=original_data_dir,
+            dataset=dataset,
+            dataset_info=dataset_info,
+            overwrite=overwrite,
+        )
+
+        basepath = "https://webdav.aeronomie.be/guest/o3_cci/webdata/Nadir_Profiles/L3/IASI_MG_FORLI"
+
+        wget_options = [
+            "-e robots=off",  # ignore robots.txt
+            "--no-parent",  # don't ascend to the parent directory
+            "--accept=nc",  # download only *.nc files
+            "--user=o3_cci_public",  # user name
+            "--password=",  # empty password (no password needed for public access)
+        ]
+
+        loop_date = start_date
+        while loop_date <= end_date:
+            year = loop_date.year
+
+            # directory on server to download
+            remotepath = f"{basepath}/{year}"
+            downloader.download_folder(remotepath, wget_options)
+
+            loop_date += relativedelta.relativedelta(years=1)
+
     else:
-        raise ValueError(f"Unknown dataset: {dataset}")
+        errmsg = f"Unknown dataset: {dataset}"
+        raise ValueError(errmsg)