Intake-ESM datastore for the ACCESS-ESM1.6 spin-up / piControl runs

[rbeucher]

Hi @charles-turner-1,

As we discussed today, it would be great if you could set up a datastore for the piControl run experiments used by the ESM1.6 development team.

see example here: /scratch/p66/pjb581/access-esm/archive/pi_concentrations-expt-c55f7217

Ideally, we’d want something flexible—maybe a command-line tool that can generate a new datastore on demand. The model outputs are evolving, and it’s getting tricky to keep track of what’s available. Having an intake datastore should make it much easier to explore the data.

Thanks!

[charles-turner-1]

$ build-esm-datastore --help
usage: build-esm-datastore [-h] [--builder BUILDER] [--builder-kwargs [BUILDER_KWARGS ...]] [--expt-dir EXPT_DIR]
                           [--cat-dir CAT_DIR] [--datastore-name DATASTORE_NAME] [--description DESCRIPTION]

Build an esm-datastore by inspecting a directory containing model outputs. If no datastore exists, a new one will be
created. If a datastore exists, it's integrity will be verified, and the datastore regenerated if necessary.

options:
  -h, --help            show this help message and exit
  --builder BUILDER     Builder to use to create the esm-datastore. Builders are defined the source.builders module.
                        Currently available options are: AccessOm2Builder, AccessOm3Builder, Mom6Builder,
                        AccessEsm15Builder, AccessCm2Builder. To build a datastore for a new model, please contact the
                        ACCESS-NRI team.
  --builder-kwargs [BUILDER_KWARGS ...]
                        Additional keyword arguments to pass to the builder. Should be in the form of key=value.
  --expt-dir EXPT_DIR   Directory containing the model outputs to be added to the esm-datastore. Defaults to the
                        current working directory. Although builders support adding multiple directories, this tool
                        only supports one directory at a time - at present.
  --cat-dir CAT_DIR     Directory in which to place the catalog.json file. Defaults to the value of --expt-dir if not
                        set.
  --datastore-name DATASTORE_NAME
                        Name of the datastore to use. If not provided, this will default to 'experiment_datastore'.
  --description DESCRIPTION
                        Description of the datastore. If not provided, a default description will be used:
                        'esm_datastore for the model output in {--expt-dir}'

We already have this available - is that what you were thinking?

[charles-turner-1]

Ahh - we need an ESM1.6 builder for this it seems.

The CM2 builder is essentially an alias for the ESM1.5 builder with some extra regex patterns, and we've got an open issue for the CM3 builder: #168.

Would I be correctly assuming that the ESM1.6 & CM3 builders are going to be similar?

[dougiesquire]

@charles-turner-1 I think it will be pretty close to the ESM1.5 builder