[Proposal] CLI & packaging update

[jasonb5]

I'm circling back to this, which was originally part of the larger refactor effort.

The proposal is to add a new CLI to CIME and start packaging it via PyPI and conda, providing a modern way to interact with and manage CIME.

Formalizing an API

CIME does not currently have an official API. While it’s possible to write Python code using the package, this is not straightforward.

Formalizing an API would encourage this use case and enable the creation of packages that utilize CIME. Packaging via PyPI and conda would further support this.

One idea is a tool that could manage cases for a user. While CIME already supports ensembles, this could be extracted into its own tool. Doing so would help keep CIME leaner and reduce feature creep. A possible vision for this tool would be a single YAML file describing the configuration for all ensemble members, paired with a single command to create, build, submit, and monitor the runs.

Packaging

Utilizing CIME as a sub-module will still be supported (unless determined otherwise).

CIME will adopt continuous delivery and utilize SemVer to track API and backward compatibility changes. This will allow models to use the version constraining features of pip and conda, providing a more reliable way to track compatible CIME versions.

The packages will also provide users with new ways of working with CIME, whether by calling Python code directly or using it as a dependency.

CLI

The CLI will condense all of CIME's tools into a single entrypoint. This will also remove the need for symlinked tools in a case directory, making cases more portable.

A potential new feature could be the ability to export/import a case. Exporting would capture a case while excluding build/run directories and machine-specific configuration. Importing would recreate the case and apply the appropriate machine-specific details.

The CLI could look like:

cime config show
cime config set

cime create_newcase
cime create_clone
cime create_test

cime case.setup
cime case.build
cime case.submit
cime xmlquery
cime xmlchange

Or in a cleaner form:

cime config show
cime config set

cime create
cime clone
cime test

cime setup
cime build
cime submit
cime preview
cime query
cime change

To support this, CIME will need to change how it finds a model's configuration, another step in decoupling the model from CIME. The config_files.xml file will need to be completely migrated from CIME to the model repositories. Then, CIME will implement a new method to determine a model's configuration by locating the correct cime_config in a model’s repository.

CIME will be able to find the source for a case using SRCROOT. A new CIME_VERSION variable will also be added to store the version used to create the case. These variables may then be used to set up the model source and CIME version from an imported case.

Feedback and suggestions are very welcome.

[jedwards4b]

This looks like a reasonable plan. To what extent do you think we can do this in incremental steps? For example, I can imagine a first step being moving the config_files.xml file out of cime into the model repositories.

[bartgol]

I like the plan. Some thoughts:

• the 2nd CLI form is much nicer
• Pinging the env for a SRCROOT var is fine, but I would also consider using cwd if possible (e.g., if cwd is a git repo and $reporoot/cime_config/config_files.xml exists)
• Are you planning on supporting both CLI usage as well as usage from within a py script? I would love run scripts to move from bash to py...
• I would very much welcome YAML config settings, although XML is more flexible in that you can use node attributes to provide selection criteria for mach, compset, grid, etc. But YAML is, imho, very easy to read (which helps a lot new users), so if configs files are created/changed, I would encourage at least thinking about it.

And I agree with Jim on moving config files to the model repo as a first step.

[billsacks]

Thanks a lot for starting this discussion, @jasonb5 !

I like the idea of formalizing a CIME API. I have leveraged CIME code from other places and it would be great to have a more formalized way to do this.

I'd like to have more discussion around the CLI. I think I'm missing something about how this would identify the correct cime version/path to use for a particular model version (for commands like create_newcase) or for a particular case (for commands like case.setup or xmlquery). You mention a CIME_VERSION variable, but I think I'm missing something about how we can avoid some circularity here: it seems like the appropriate cime version needs to be loaded into your path before running any cime commands, so by the time you have run cime ... it is too late to read CIME_VERSION....?

I'd also like to have more discussion around the packaging ideas, specifically in terms of tracking CIME version compatibility with different model versions. What you're suggesting makes a lot of sense in terms of making CIME development work more like typical python packages. And yet, this is a departure from the way we've grown accustomed to working, where there is often co-development between CIME and other repositories using CIME that means we do not need to work as hard to maintain backwards compatibility in various respects - which is possible due to the exclusive use of CIME as a submodule, which means that we can tightly control the version of CIME included in any model and ensure compatibility. There are pros and cons of the different approaches, but I just want to be sure that we're thinking through the potential costs and making sure that the benefits feel like they justify those costs.

My overall view on this is that the formalization of a CIME API, and building out CIME's capabilities as a python library that could be leveraged by other tools, could be very valuable. And I see that doing that pushes us towards having CIME be developed, maintained and distributed more like a standard python package. But I'm still struggling somewhat to understand the rationale and benefits for the change to the CLI and the potential changes to how we ensure compatibility between CIME and model versions. It may just be that I haven't wrapped my head around this yet and need some more enlightenment 😊. I'm open to being shown these benefits, but am just approaching this somewhat conservatively because of the big potential changes this would have for both users and developers. Most importantly, though, I really appreciate your raising this: these are really great ideas to consider.

It seems like we might be able to make faster progress towards some shared vision / agreements here via a meeting. Should we reboot the dormant CIME calls to discuss this further?

[bartgol]

I think packaging cime as a pypi package would have a few benefits:

• deps pkgs (with their version too!) tracked by pip: no need for the host model to make sure that all modules needed by CIME are installed; just run pip install cime=1.2.3 and be done.
• since pip works with --user, one can run pip install --user cime and get all the deps of CIME even if they are not installed on that machine and they have no sudo powers, without having to spend time dealing with this stuff
• if a new version of CIME doesn't work, simply run pip install cime=0.1.2 --force-reinstall. This may not be too different from changing the submod to point to a prev tag, but can be done once.
• we can keep the cime submodule too, so if users don't want to use the installed pip pkg, they can still do ./cime/scripts/create_newcase ...
• while we can maintain a py-venv/conda-env on our tgt machines, users are free to run pip install --user cime, which will be preferred to the system one (even if the "e3sm-supported-conda-env" is still loaded). So if my branch has some issue, I can quickly experiment with an old cime without touching the repo or the system-wide env
• other python tools (postproc, but also others) could use some cime capability without having to clone CIME. E.g., I may want to get the env mach specific for the curr machine, so that I can do some stuff. Simply from cime import get_case_env and use it
• probably easier to try new cime versions in CI jobs if we can just add cime to the venv setup step, rather than have to checkout a different submodule version.

[jasonb5]

@jedwards4b @bartgol @billsacks

Appreciate the feedback.

This will be done in steps to ensure we don't disrupt work. Usage as a sub-module will continue unimepeded.

• The first step will moving the CESM config files. I'll put togther the additional steps we'll take.
• I'm in favor the of the 2nd CLI form as well, much cleaner.
• As for detecting the models root directory, we'll define a very clear and concise hierachy. As of right now I'm thinking user input via --srcroot to commands, env variable SRCROOT, falling back to checking cwd for a git directory.
• The goal will be to support CLI and py script. Having a formal API will go a long way with usability within python files.
• Implementing YAML is something I've been considering for a while. While it does lack the attribtues concept, there are other methods. For instance JSON is a subset of YAML, and we have acccess to libraries like jsonpath that can be used to query similar to XQuery. If we don't want to introduce third party libraries it wouldn't be hard to filter using KV pairs. This is something still being considered but out of scope for this effort.
• Concerning the CLI matching CIME/model version. Initially it's up to the user to install the correct CIME version for the model using either a pypi or conda requirement document if supplied by the model. The documents will be how the model can track the compatible CIME version. Once a user invokes a CLI command and CIME determines the model or case root directory, it can use either the requirements doc or CIME_VERSION variable respectively to determine compatibility. The CIME_VERSION only concerns cases, this is how a case tracks compatible CIME version.
• There could be some loss of granularity in relation to the model specifying the compatible CIME version. Right now the model can define this down to the commit hash where moving to packages it may only define this to a version containing the specific change. There's definitely the question of whether this granularity is required. Even moving to packages using semver we could consider each patch bump not compatible with the previous though this isn't the correct usage of semver and I don't know if we want to publish that maybe package releases.
• The CLI brings some mobility to the tool e.g. I can run commands from anywhere. It would remove the symlinks from case directories, making them more concise, a case directory now only contains what describes the case. This also benefits adding an ability to export/import cases between machines. Right now we have scripts and tools under the CIME root which are not clear where they can run from or what they can run on without diving into the documentation. The CLI cleans this up by being self describing (examples differ from above) cime create to create a case, cime create --clone to clone a case, cime test to run a test, cime case setup to run setup in a case directory. The CLI works hand-in-hand with packages, @bartgol listed benefits of the package. An additional benefit of packages we can pin the compatible versions of python.

I'm open to reinstating the CIME call to discuss this further.

@rljacob @jgfouca Would be grateful to hear your thoughts.

[jedwards4b]

Thank you for your work on this Jason!

[billsacks]

I appreciate your detailed comments @bartgol and @jasonb5 . I'm still not sold on the idea of having cime versions managed separately from the model source code because of the common scenario where a single person is working on projects with multiple model versions, which could each need their own version of cime. I'm still not clear on how that would be handled in practice. But I'm happy to be part of more discussions on this that could help me see how this will work. I do see the benefits of a lot of what you're proposing for certain use cases, especially where CIME can be leveraged for things outside of just the case control system, so maybe the solution to my concerns is just to plan to continue the usage as a submodule for the sake of case control system-like usages.

[jedwards4b]

@billsacks wouldn't you just have a different conda environment for each model version you were working with?

[samsrabin]

I have the same reservation as @billsacks. A separate conda environment for each model version is certainly possible but in practice will be a pain. This would be moderated to some extent if new developments try as much as possible to maintain backwards compatibility, and of course you've mentioned that "CIME as submodule" would still be supported... But that does start to erode the "help keep CIME leaner" goal of this effort.

One thing I would like to see usability perspective is good error messaging. If a user tries doing something with an incompatible version of CIME, there should be an instant error message like "This version of the model requires cime version $minimum_version or later; you're using $active_version." (As opposed to some Python traceback for an unhandled exception.)

There's also the issue that conda environments aren't great to have to live in. I know that @mvdebolskiy has complained about CTSM's reliance on a custom conda environment because they're a big problem on the NorESM machine. For machines like that I guess it'd be up to the administrators to make the environments available somehow, e.g., through module or spack.

Even on Derecho, I feel like sometimes my user-space conda environment is really slow to start some commands. My assumption there is that it is getting unloaded from cache because I'm the only one using it? I don't really know what I'm talking about, so that might be off base.

All that said, if one result of this is that we can start to use YAML or something similarly modern for our configuration files, it might all be worth it! Only partially joking!

[bartgol]

Do you really foresee different model versions needing different CIME versions? We have hit the case where we had to revert the CIME submodule to a prev version b/c some changes broke e3sm, but a) that wasn't very common, and b) maybe points to the need for a better integration testing for CIME (where CI jobs of downstream repos are triggered with the new CIME version before the PR is merged).

I agree that if each model version needs a different cime pkg version then this approach is not optimal. But isn't usually ok to just use the latest CIME?

Again, if CIME updates are constantly (or at least "too often") breaking a downstream model, then we should prob first invest into adding downstream model tests to the CIME pr CI system...

[jgfouca]

Do you really foresee different model versions needing different CIME versions?

@bartgol , yes.

There are a couple scenarios:

1. A model is using a CIME feature that has been changed or removed. We have a github label backwards incompat that we are supposed to use for these changes. I only see a few PRs with this label, but I think we forgot to put it on some PRs.
2. A model wants to use a newer CIME feature that only exists in a recent version of CIME. This is very common: a model requests a feature, we add it to CIME master, and the model updates their subm to get it.

[bartgol]

Well, option 2 does not mean you can't use new CIME with an old model. Only if the new option is not bwd compatible (like in 1). If it is not interfering with the old CIME functionalities, customers may just pip upgrade cime on their platform, and use it with both new and old model versions.

In e3sm, I did see CIME updates that broke e3sm. But from what I could tell, it was mostly b/c the feature was not really fully tested with e3sm before being integrated, no? Hence, my comment about maybe mitigating disruption by increasing integrated cime+e3sm and cime+cesm CI testing...

(I may be naive here, as I am not following closely the e3sm+cime integration soap opera)

[samsrabin]

Better test coverage of model-CIME interaction is definitely important, and if this effort helps us improve that it will definitely be a good thing. I feel like our CTSM testing (CESM land model) is pretty thorough, but it's always hard to know for sure, and earlier this summer we found a pretty major CIME-related bug that our testing had missed. The challenge, as with everything, is that it's hard to find person-power for improving test coverage.

I'll also note that even CIME itself doesn't have great coverage, with a recent codecov report showing only 55% coverage. Plus, even 100% coverage in terms of lines touched doesn't necessarily mean all the different possible code paths have been covered.

[bartgol]

@samsrabin What I was envisioning is CIME's pr CI to trigger some downstream repo CI job on a particular branch. Something like this:

• CIME PR is open
• CIME's CI creates a branch in e3sm (say e3sm-cime-update-pr-XYZ), where cime submodule is updated to the branch of the current CIME PR
• CIME's CI triggers the e3sm's repo workflow e3sm-gh-ci from the branch e3sm-cim-update-pr-XYZ
• CIME's CI waits for the e3sm's repo workflow to complete, and reports pass/fail

This pattern would guarantee that e3sm+newCIME would work (or pinpoint the e3sm needs that the current PR breaks). It would also be nice b/c CIME does NOT need to know what tests the e3sm workflow runs (they may not even use CIME, for all we know). It just has to know what the workflow name is, and trigger it. Hence, CIME devs do not need to do much work, other than setting these workflow up once.

I think gh does allow this, but there may be some requirements (e.g., the CIME repo may have to store a token from the e3sm repo, to allow triggering a workflow remotely, and monitor it). I can investigate this.

Edit: obv, I mentioned "e3sm", but this holds for any downstream repo that uses CIME and offers gh CI testing workflows.

[samsrabin]

Yes, that would definitely be useful!