Documenting the wonderful world of CMIP CVs, CMOR tables and data request

[znichollscr]

We will need some guidance on CVs, data request and CMOR tables. I know there are individual issues for these pieces (sorry I didn't find and link them), but I think these come as a bit more of a package so here are some notes on the set up as I see it now, in the hope that this will guide discussion and also be helpful for anyone trying to follow along with this work.

As I understand it, our guidance at present is:

• if you want anything to do with the data request, then you should use the data request API. The recommendation is always to use the latest version
  • this also includes anything to do with variable definitions. There is a plan to split out a variable registry in the medium-term, but for the short-term that will not be available so all variable information should be retrieved via the data request API
• if you want anything to do with CVs, you should use esgvoc, esgvoc docs
  • this means allowed values for global attributes, definition of the DRS etc.
  • esgvoc also defines schemas for all data descriptors used within CMIP, they can be found in https://github.com/ESGF/esgf-vocab/tree/main/src/esgvoc/api/data_descriptors. These schemas are currently being refined and updated. We will update this issue when they are finalised. They are being updated in line with this spreadsheet (https://zenodo.org/records/17250297), although changes during this translation process are likely so please take that spreadsheet as a first, albeit relatively stable, draft rather than the final word
  • the source data for esgvoc comes from the esgvoc of various CVs repositories (https://github.com/WCRP-CMIP/WCRP-universe/tree/esgvoc, https://github.com/WCRP-CMIP/CMIP7-CVs/tree/esgvoc, https://github.com/WCRP-CMIP/CMIP6_CVs/tree/esgvoc). While you can obviously see this source data, please use esgvoc to access the information rather than trying to parse the schemas yourself. Parsing the schemas is real effort: esgvoc has made this effort so you don't have to
• in global attributes, we will include a data_specs_version that will identify different releases of esgvoc and the data request API. We are still exploring exactly how this will be defined. We will update this issue as we make progress on this. For now, feel free to just use the placeholder value "0.0.0"
• if you want CMOR tables, they are in the process of being produced. The variable tables will be produced via the data request API. The CVs JSON file will be produced via an 'esgvoc app'. The outputs of both these processes will be made publicly available. We will update this issue with details as they solidify
  • if you would like to see an early prototype of exporting the variables tables from the data request, see https://gist.github.com/matthew-mizielinski/3103beb41e2b83a51b7ae923592489cf
  • if you would like to see an early prototype of exporting the CVs table from esgvoc, see Demo do not merge: add exporting of CMOR CVs table ESGF/esgf-vocab#51

[matthew-mizielinski]

We will need some guidance on CVs, data request and CMOR tables. I know there are individual issues for these pieces (sorry I didn't find and link them), but I think these come as a bit more of a package so here are some notes on the set up as I see it now, in the hope that this will guide discussion and also be helpful for anyone trying to follow along with this work.

As I understand it, our guidance at present is:

• if you want anything to do with the data request, then you should use the data request API. The recommendation is always to use the latest version

Correct

• this also includes anything to do with variable definitions. There is a plan to split out a variable registry in the medium-term, but for the short-term that will not be available so all variable information should be retrieved via the data request API

Correct

• if you want anything to do with CVs, you should use esgvoc, esgvoc docs

Correct, as long as there can be some clear guidance on using this for CMIP7,

• this means allowed values for global attributes, definition of the DRS etc.
• esgvoc also defines schemas for all data descriptors used within CMIP, they can be found in https://github.com/ESGF/esgf-vocab/tree/main/src/esgvoc/api/data_descriptors. These schemas are currently being refined and updated. We will update this issue when they are finalised. They are being updated in line with this spreadsheet (https://zenodo.org/records/17250297), although changes during this translation process are likely so please take that spreadsheet as a first, albeit relatively stable, draft rather than the final word
• the source data for esgvoc comes from the esgvoc of various CVs repositories (https://github.com/WCRP-CMIP/WCRP-universe/tree/esgvoc, https://github.com/WCRP-CMIP/CMIP7-CVs/tree/esgvoc, https://github.com/WCRP-CMIP/CMIP6_CVs/tree/esgvoc). While you can obviously see this source data, please use esgvoc to access the information rather than trying to parse the schemas yourself. Parsing the schemas is real effort: esgvoc has made this effort so you don't have to

Fine as long as we are happy that esgvoc tools are returning CMIP7 information.

• in global attributes, we will include a data_specs_version that will identify different releases of esgvoc and the data request API. We are still exploring exactly how this will be defined. We will update this issue as we make progress on this. For now, feel free to just use the placeholder value "0.0.0"

I'd prefer something other than "0.0.0" if we can, but I don't know what this should be.

• if you want CMOR tables, they are in the process of being produced. The variable tables will be produced via the data request API. The CVs JSON file will be produced via an 'esgvoc app'. The outputs of both these processes will be made publicly available. We will update this issue with details as they solidify

With the revised version of the MIP tables we can point at this later this week. For the CMIP7_CVs JSON file I think we should host this in the CVs repo. It would be useful to have an updated example to go here.

• if you would like to see an early prototype of exporting the variables tables from the data request, see https://gist.github.com/matthew-mizielinski/3103beb41e2b83a51b7ae923592489cf

Examples in the mip tables -- shortly to be moved to https://github.com/WCRP-CMIP/cmip7-cmor-tables

• if you would like to see an early prototype of exporting the CVs table from esgvoc, see Demo do not merge: add exporting of CMOR CVs table ESGF/esgf-vocab#51

[znichollscr]

Correct, as long as there can be some clear guidance on using this for CMIP7

There definitely can be, although isn't any yet

I'd prefer something other than "0.0.0" if we can, but I don't know what this should be.

0.1.0 ? :) (anything pre v1 is 'pre-release' in semantic versioning land so fair game I think)

With the revised version of the MIP tables

Are MIP tables variable tables? Or does MIP table have a different meaning?

For the CMIP7_CVs JSON file I think we should host this in the CVs repo. It would be useful to have an updated example to go here.

Sounds good