Help and Guidance for CESM Submodule Updates

Help and Guidance for CESM Submodule Updates

Introduction

The .gitmodules file contains the submodules that are needed for CTSM to run. This list include the CESM submodules:

• cime
• cdeps
• cmeps
• ccs_config_cesm
• share
• mpi-serial
• parallel-io

As well as CESM subcomponents such as:

• RTM
• MOSART
• mizuRoute
• CISM

Sometimes a specific submodule can be updated independent of the others. But, sometimes there is a dependency between two or more of them. When this happens it can be tricky to figure out a set of submodules that all work together. Sometimes that interdependency is documented in the CESM testdb for either the planned tags or for existing tags. And sometimes the commit log will document these interdependencies, but not always. The version number for a submodule may also bump up the major or minor version for it to denote such interdependencies.

Suggestion 1 -- always look at CESM alpha tags

Go to the CESM testdb and look at the alpha tags (both planned and completed ones).

https://cseg.cgd.ucar.edu/testdb

For completed tags, this shows you the submodules that were tested together.

Suggestion 2 -- look for dependencies in the testdb

As part of looking at the testdb from above, look to see which alpha tags document dependencies between submodules. Looking at the description in testdb or the names of PR's in github might also indicate when tags between submodules had to be coordinated. Sometimes PR's might document what externals where used together to make them.

Suggestion 3 -- try to narrow down a problem to one submodule causing it (or at least as few as possible)

If you can narrow down a problem to one submodule, you can then use git bisect or run through the tags for that submodule to get down to the problem. An important part of this is having a point that works and a point where it doesn't. Then you can track down to the tag or commit with the problem. You also might be able to see the problem by looking at the differences, but if there are too many changes -- it might not be easy to spot. And you'll need to narrow the range to make it more obvious. Part of narrowing it down to one submodule is to make a guess at what the problem is. A build issue might be ccs_config or cime. A driver issue CMEPS, a data model issue -- CDEPS. Problem with running tests or cases will likely be cime.

An example of filling out a table for this purpose where there is one submodule causing the problem:

https://github.com/ESCOMP/CTSM/pull/3111#issuecomment-2855578097

Suggestion 4 -- what about when there are multiple submodules?

If you can't narrow it down to one submodule -- narrow it down to as few as possible. And then use tags to try combinations until you reach a point where you see a small range of tags between working and failing.

Here's an example of a case where the problem involves multiple submodules, and narrowing it down. In this case alpha tags are used as a starting point as a way to test versions expected to work together. Also examining the history of .gitmodules in CESM can be helpful as well (see next suggestion).

https://github.com/ESCOMP/CTSM/pull/3125#issuecomment-2869212442

Suggestion 5 -- use the history of .gitmodules in CESM

The history of the .gitmodules file in CESM might show points that were tested that were in-between alpha tags. And may give something useful to try.

Suggestion 6 -- debug the problem in the submodule that's not working

Normal, debugging the problem happens when other methods aren't helping. You also should file an issue in the repository with the problem and ask for advice in it's code maintainers. And of course bring in others to go over the problem when you are stuck. Even just explaining the problem to someone else can cause you to slow down your thinking and present it in a logical way that enables you to figure it out. That last idea is called "rubber duck debugging".

Suggestion 7 -- look at CESM nightly testing with latest submodules

Nightly tests appear on Derecho here (it runs the aux_cime_baselines testlist):

/glade/derecho/scratch/fischer/nightly

With the date of the test as MMDDYYY and with the tests run

The code being tested is here:

/glade/u/home/fischer/code/cesm3_nightly_tests