Unsuccessful searches across nodes due to ORNL node not working

[jlenh]

In ESMValTool, the searches on ESGF nodes are being done using the following base configuration for esgf-pyclient.
However, testing on CMIP5 data from the abrupt4xCO2 experiment, while files are available on other nodes (e.g. CEDA), the search on the ORNL node (which is currently not working as of 18.08.25) overrides any results from other nodes, and results in no data being downloaded at all.

Extract from https://github.com/ESMValGroup/ESMValCore/blob/main/esmvalcore/config/_esgf_pyclient.py

cfg = {
    # Arguments to
    # https://esgf-pyclient.readthedocs.io/en/latest/api.html#pyesgf.search.connection.SearchConnection
    "search_connection": {
        # Be careful about the url, not all search urls have CMIP3 data?
        "urls": [
            "https://esgf-node.ornl.gov/esgf-1-5-bridge",  # Report any issues here: https://github.com/esgf2-us/esg_fastapi/issues
            "https://esgf.ceda.ac.uk/esg-search",
            "https://esgf-data.dkrz.de/esg-search",
            "https://esgf-node.ipsl.upmc.fr/esg-search",
            "https://esg-dn1.nsc.liu.se/esg-search",
            "https://esgf.nci.org.au/esg-search",
            "https://esgf.nccs.nasa.gov/esg-search",
            "https://esgdata.gfdl.noaa.gov/esg-search",
        ],
        "distrib": True,
        "timeout": 120,
        "cache": "~/.esmvaltool/cache/pyesgf-search-results",
        "expire_after": 86400,  # cache expires after 1 day
    },
}

Example of request:

• GET /esgf-1-5-bridge/?format=application%2Fsolr%2Bjson&limit=500&distrib=true&offset=0&type=File&project=CMIP5&model=GFDL-CM3&ensemble=r1i1p1&experiment=abrupt4xCO2&time_frequency=mon&institute=NOAA-GFDL&cmor_table=Amon&product=output1%2Coutput2&variable=rlds HTTP/1.1
• Found the following files matching facets {'project': 'CMIP5', 'model': 'GFDL-CM3', 'ensemble': 'r1i1p1', 'experiment': 'abrupt4xCO2', 'time_frequency': 'mon', 'institute': 'NOAA-GFDL', 'cmor_table': 'Amon', 'product': 'output1,output2', 'variable': 'rlds'}: none

[bouweandela]

The issue reported here seems to be that no files are found with

https://esgf-node.ornl.gov/esgf-1-5-bridge/?format=application%2Fsolr%2Bjson&limit=500&distrib=true&offset=0&type=File&project=CMIP5&model=GFDL-CM3&ensemble=r1i1p1&experiment=abrupt4xCO2&time_frequency=mon&institute=NOAA-GFDL&cmor_table=Amon&product=output1%2Coutput2&variable=rlds

while e.g.

https://esgf.ceda.ac.uk/esg-search/search?format=application%2Fsolr%2Bjson&limit=500&distrib=true&offset=0&type=File&project=CMIP5&model=GFDL-CM3&ensemble=r1i1p1&experiment=abrupt4xCO2&time_frequency=mon&institute=NOAA-GFDL&cmor_table=Amon&product=output1%2Coutput2&variable=rlds

returns 30 files.

Is the expectation that the Globus index is synchronized with the Solr indices?

[nocollier]

The 1.5 bridge Globus-based index was created from all records that were on a US Department of Energy Solr index node. That is primarily all the records that were previously on the LLNL node plus some more replicas at some other labs. We did this to try to transition much of ESGF off of Solr as best we could, but we do not have control of the other centers' index nodes and hitting them with traversal requests through esg-search was deemed not kind or feasible.

I do not think we plan to leave CMIP6 partially off Solr indefinitely. My impression from meetings is that we plan to also move CMIP6 into a STAC catalog eventually, but I know of no time table currently.