Question: cmip7 - cmip6 mapping

[treerink]

Hi,

I am trying to find access to the mapping between the cmip7 - cmip6 compound variables. I am interested to find a piece of python dreqCMIP7 code in which I can iterate over the CMIP7 compound variables and then to find for each CMIP7 compound variable its CMIP6 table - variable combination (for all those which do have an equivalent). And if possible also for the opposite direction: CMIP6 => CMIP7.

For instance in the export_dreq_lists_json.py script I have been looking into the dc.mapping_table with:

     for k, v in dc.mapping_table.items():
      print("key: ", k, "; value: ", v)

I see for instance CMIP6 Frequency (legacy), but I doubt this is the easy way to go.

I read from the README.md:
Each listed variable in the output file is currently identified by a unique "compound name" using CMIP6-era table names and short variable names (Amon.tas, Omon.tos, etc). Variable names may change in upcoming releases, but in any case a mapping to CMIP6-era variable names will be retained in the data request so as to allow comparison with CMIP6 output (for those variables that were defined in CMIP6).

We have our own mapping of EC-Earth3 against CMIP6 about what we are able to deliver. In order to reuse this to a large extent it would be very helpful to have a few lines code to do this iteration and CMIP7 - CMIP7 mapping. Hoping for some lines of code which can do this, any suggestion?

Thanks in advance
Thomas

[JamesAnstey]

@treerink Thanks for this question, and sorry for not providing a response earlier here in this issue. This is an important question for data producers. I'll try to expand here on what was said in the v1.2 launch session about an hour ago. Currently the mapping from CMIP6 compound name (e.g. "Amon.tas") is provided by scripts/examples/variables_1.2.1.json, which lists variables keyed by their CMIP6 names and for each one gives the branded variable name, for example:

        "3hr.clt": {
            "frequency": "3hr",
            "modeling_realm": "atmos",
            "standard_name": "cloud_area_fraction",
            "units": "%",
            "cell_methods": "area: time: mean",
            "cell_measures": "area: areacella",
            "long_name": "Total Cloud Cover Percentage",
            "comment": "for the whole atmospheric column, as seen from the surface or the top of the atmosphere. Include both large-scale and convective cloud.  This is a 3-hour mean.",
            "dimensions": "longitude latitude time",
            "out_name": "clt",
            "type": "real",
            "positive": "",
            "spatial_shape": "XY-na",
            "temporal_shape": "time-intv",
            "cmip6_table": "3hr",
            "physical_parameter_name": "clt",
            "variableRootDD": "clt",
            "branded_variable_name": "clt_tavg-u-hxy-u"
        },

So 3hr.clt --> clt_tavg-u-hxy-u, etc. As @bethdingley mentioned in the launch session the branded names in dreq v1.2.1 are provisional pending finalization of the branding algorithm (but should be finalized for dreq v1.2.2), and also branded_variable_name does not uniquely defined a requested variable because information on the frequency and region is also needed. (Most requested variables are global, but not all, hence region is needed.)

We would like to provide the mapping between new and old names in as transparent a way as possible, so that data producers can best use their existing tools (that may have some dependence the old names) and get familiar with the new names (which will be used in the directory and filenames of netcdf produced by CMOR).

A simple way to provide the mapping is a dictionary (json file) giving the one-to-one conversion between old and new names (where for this purpose the new name needs to include frequency and region to ensure uniqueness). We could also make this an API function, using the same mapping under the hood but maybe more convenient for users. Happy to hear any feedback on how this could best work.

[treerink]

@JamesAnstey thanks a lot, with your comment I have been able after some more studying to find my way in a better direction. In principle I am now able to load the necessary stuff in python and to iterate over the variable list (the compound variables) requested by each experiment and to detect for each variable the cmip6_table and the physical_parameter_name (I guess that one is always identical to the cmip variable name?), and with the cmip7 branded_variable_name. For this I am now using an adjusted version of the export_dreq_lists_json.py script.

However, I wondered whether it would be possible to add to the get_variables_metadata function in CMIP7_DReq_Software/data_request_api/data_request_api/query/dreq_query.py also the option to select on a cmip7 experiment request (or list of experiments)?

So in general I think a commandline tool for the mapping and a way to iterate over requests and then to have the mapping available in python scripts by import the python cmip7 data request package is what is handy for users. I more or less have this now, but I am not sure whether wat I did is the most efficient and most clean direct way (to the base source the air tables for instance).

I noticed different results when obtaining the metadata (when it commes from the data request content side I have the impression it comes the most direct from the new cmip7 side, but I see much more info missing or flaws I think. When it comes via the export_dreq_lists_json.py script side the metadata looks much better, but I have the impression then actually a lot of the gaps are filled in with cmip6 cmor table data for the time being?

[treerink]

It would be convenient if in the function get_requested_variables of the file data_request_api/data_request_api/query/modified_dreq_query.py the print statements with Retaining only these would be removed or that an additional optional argument verbose is added to the function get_requested_variables such that these messages can be switched off. Of course I did this in a locally checkout of the CMIP7_DReq_Software but I have to use the pip install of the CMIP7-data-request-api inside another mamba environment in which I do a call to the function get_requested_variables for the cmip6 - cmip7 mapping.

Another non related tiny minor point:
In the same file in the function show_requested_vars_summary the following change get a more readable overview:

-        print(f'  {expt} : ' + ' ,'.join(['{p}={n}'.format(p=p, n=d[p]) for p in priority_levels]) + f', TOTAL={n_total}')
+        print(f'  {expt:33} : ' + ' ,'.join(['{p}={n:4}'.format(p=p, n=d[p]) for p in priority_levels]) + f', TOTAL={n_total:4}')

with more results, i.e. more experiments.

Last question, I guess that the pip package is only updated with a new tagged version release? Or is the pip CMIP7-data-request-api package automatically updated with every master commit (say once a day)?

[JamesAnstey]

Hi @treerink - sorry for my slow reply. The pip package is only updated with releases (tags) but you can do a developer install as described in the README to work with the very latest.

Agree about a verbose option for get_variables_metadata - it's added in this PR.

Thanks for the readability suggestion for dq.show_requested_vars_summary(), I'll add that or similar.

To get variables' metadata for the variables that are requested for a particular set of experiments, you can do:

export_dreq_lists_json v1.2.1 out.json --all_opportunities --experiments historical --variables_metadata vars_historical.json

where the vars_historical.json output file will like the one produced by get_variables_metadata except that it's limited to the variables requested for (in this example) the historical experiment, i.e. the variables listed in the out.json file. Is that what you need? (Right now it produces verbose stdout, but the above PR fixes that.)

Thanks for your thoughts on the mapping, which we hope to implement in the API soon.

I'm not sure what you meant by "I noticed different results when obtaining the metadata" - different results from which tools? The CMIP7 data request started with the collection of CMIP6 variables as the pool of available variables to be requested, but the metadata of some has changed and new variables have been added. Could you clarify? Btw there is a command line tool in the software, compare_variables, that might help if you need to check for changes in variable definitions (e.g., what metadata has changed in CMIP7 compared to the CMIP6 version of a variable).