Can checking out a new version affect already-submitted jobs?

[samsrabin]

(Submitting as an idea discussion rather than an issue because it's possible I'm misunderstanding what's going on.)

I have two CTSM branches with which I'm performing runs; let's call them branchA and branchB. branchB has some new namelist parameters that branchA is missing. If I submit a branchA run (or a new segment is submitted), then check out branchB before the branchA run begins, branchA fails (and without a helpful error message):

 Attempting to initialize the land model .....

 Attempting to initialize run control settings .....
 Read in clm_inparm namelist from: lnd_in
 ENDRUN:
 ERROR: ERROR reading clm_inparm namelistERROR in controlMod.F90 at line 325

The run goes successfully after I check out branchA again, resubmit, and don't check out branchB.

I'm noticing this because of the failure, but I think it's possible for unexpected differences in behavior to happen without a failure. Maybe this could happen if default namelist options differ between the two branches? Not sure.

As far as I can tell, the only way to avoid this possibility is to have two separate checkouts of the model codebase, one for each branch, which is annoying to manage. It would be better if all the files needed by a run were copied to its build or run directory during case.build so that, after building, the run is completely independent of the model source.

[samsrabin]

(For future personal reference: branchA is yield_perharv2 commit 95c6bbe; branchB is rx_crop_calendars2 commit d255bf6.)

[jedwards4b]

@samsrabin we made a design decision several years ago that a case could not be made completely independent of source as you suggest. The solution is to manage two separate source trees.

[samsrabin]

Okay, in that case I'll trust there are good reasons. Is simply copying the existing source directory to a new one okay, or is a completely new checkout needed to avoid problems?

[jedwards4b]

You can copy the existing source directory and then switch to the new branch.

[samsrabin]

Great, thanks! Feel free to close this discussion if you'd like.

[jedwards4b]

But you either need to recreate the test case or change each of these xml variables to point to the new source: CIMEROOT, SRCROOT, MACHDIR

[ekluzek]

@samsrabin I changed this into a question, so I could mark it as having an answer. And then I added the wontfix label to it, as I don't see this as something that will be worked on. This makes this discussion as resolved, but also keeps it available for others to see as well.