Supporting cross platform configurations

[aidanheerdegen]

Scheduler information is platform dependent and it is defined in the config.yaml, which means a configuration is platform specific.

ACCESS-NRI needs to support configurations on multiple HPC platforms, which would mean having duplicate model congfigurations that differ only in the scheduler information.

If payu could support specifying the scheduler information for multiple platforms, the same configuration could seamlessly support multiple HPC platforms/targets.

Proposal

payu support multiple platform/site specific scheduler information, e.g.

site:
  gadi:
    project: xx99
    queue: normal
    walltime: 2:30:00
    jobfs: 1500MB
  setonix:
    project: project123
    queue: work
    walltime: 2:00:00

and/or if payu supported including yaml files

site:
  !include gadi.yaml
  !include setonix.yaml

gadi.yaml:

 gadi:
  project: xx99
  queue: normal
  walltime: 2:30:00
  jobfs: 1500MB

setonix.yaml:

setonix:
  project: project123
  queue: work
  walltime: 2:00:00

Note: this syntax could be extended to support multiple queues at a single site, but there are likely accompanying configuration changes so not sure it is that useful. It would also require some way of specifying which queue/environment was required from the command line, whereas site can be inferred automatically.

Related issues: #182 #323 #568

[jo-basevi]

ACCESS-NRI needs to support configurations on multiple HPC platforms, which would mean having duplicate model congfigurations that differ only in the scheduler information.

There's still a lot nci-specific path values in the current configs, e.g. in model inputs, module use path for executables, potentially userscripts could be some external script in /g/data, restarts. So would there be multiple site: sections nested at different levels, or a separate site configuration that gets merged into the general one?

If everything will be installed/added in a similar directory structure to /g/data/vk83, and uses the same files, we could also use jinja templating for path prefixes and scheduler-specific values?

[aidanheerdegen]

There's still a lot nci-specific path values in the current configs, e.g. in model inputs, module use path for executables, potentially userscripts could be some external script in /g/data, restarts.

You make some excellent points, thanks for the feedback.

Clearly there would need to be some other changes as you rightly point out.

So would there be multiple site: sections nested at different levels

That is certainly one approach, but it would be a bit messy for inputs at least.

Now that payu properly supports symlinks for input dirs we could use input paths to a symlink that is created in the control directory.

Creating the symlink could be created with an init user script that has some logic that checks the site.

... or a separate site configuration that gets merged into the general one?

That is interesting ...

module use path for executables

That is probably amenable to being added in a site specific file. I guess the init userscript mention above could be as well.

If everything will be installed/added in a similar directory structure to /g/data/vk83, and uses the same files, we could also use jinja templating for path prefixes and scheduler-specific values?

Yes we could. @marshallward was always opposed to templating in the config.yaml but we might be getting closer to that point. I know the ocean sea-ice team were talking about using templates to keep their dizzying number of configurations in-step.

My hesitation ... this is what is done with rose+cylc suites, and I'm not a fan of the complexity/opacity that can lead to.

[aidanheerdegen]

Rather than use symlinks we could have an inputpath option that acts like the $LABORATORY/input location, and looks for the inputs relative to that root, and define the inputs relative to inputpath.

However there are other problems:

• Manifests have fullpaths and we currently do a payu setup to write the input and exe manifests, and a commit, so the configuration reflects the correct paths on the HPC and we have reproduce: true for both. Different sites would require different manifests
• restart paths