Revise documentation on how to use a branch of a component within a CESM checkout

[billsacks]

I periodically get questions about how to point to a branch of a component (say, CAM, CTSM or CISM) within a CESM checkout (most recently from @Ivanderkelen). We have some documentation in the CESM README file, but I'm not sure that what we have is actually the method that makes the most sense. In particular, there are at least two problems with telling people to point to their branch in Externals.cfg:

1. Support for branches with manage_externals still leaves a lot to be desired (see Handle checking out a branch when that branch is already checked out ESMCI/manage_externals#34)

2. Additional problems are caused if you change an Externals.cfg file in the middle of the tree (rather than the top-level file). For example, if you want to point to a different version of FATES (an external of CTSM), you might be tempted to modify the file components/clm/Externals_CLM.cfg. However, if you do that, when you rerun manage_externals from the top level (from the root of the CESM clone), you will get an error because the components/clm external has a modified file. (@ekluzek has suggested allowing checkout_externals to proceed if there are modifications in these Externals files. This may be worth considering, although it might be hard to implement and the implications should be thought through carefully.)

So I'm inclined to recommend that people checkout their branch using regular git commands. For example, if you want to checkout your branch of CTSM in a CESM checkout, I would recommend getting CESM as normal and doing an initial run of manage_externals/checkout_externals. Then do:

cd components/clm
git remote add ...
git fetch ...
git checkout ...

However, this isn't completely straightforward if your branch is of a component that has its own sub-externals (e.g., you have a branch of CTSM, which has a FATES external). In this case, I would probably still recommend using the above procedure to get your branch, but then getting any sub-externals by running the following from within components/clm (NOT from the CESM top level):

./manage_externals/checkout_externals clm

I'd like to hear some thoughts from others. I don't feel in a rush to update this documentation, so I think it's best if we take some time to gather thoughts until we feel pretty happy with a recommendation for users.

@gold2718 @cacraigucar @nusbaume @ekluzek @Katetc @mnlevy1981 @alperaltuntas @jedwards4b @mvertens @rsdunlapiv @uturuncoglu

[cacraigucar]

I think there may not be a one-size-fits-all recommendation. I end up doing both approaches depending on my needs.

I personally start out with changing the Externals.cfg file and pointing to my branch in it. That is easy and suits my needs a good portion of the time.

At other times(listed below), I will remove (or rename) the component in question and by hand do a
git clone ...
git checkout...

The times that I retrieve the component by hand are:

• I have made mods anywhere and checkout_externals doesn't work
• I plan on making mods to my fork, so I don't have a detached head and can commit my changes easily

I suppose we could document the two approaches along with the times when a user might prefer one over the other.

[Katetc]

Hi all,

I really, really prefer to work with the Externals.cfg files. I find that they are an important way to document exactly which versions of which components you are using for your experiments. This is super important for papers and reproducibility. I think it would be very valuable to have manage_externals ignore the changes to an Externals.cfg file, so that all checkouts can be done using one method and one tool. That would be my vote.

[cacraigucar]

If we were to additionally modify manage_externals to have a flag to allow checkouts to not have detached heads, then I too could depend on it 100% of the time (assuming Kate's suggestion was also addressed).

[ekluzek]

@cacraigucar if the top level external points to a branch in a component, by default it checks it out as a detached-head, but you can change it to a branch using git commands, and manage_externals will continue to work correctly after that. Now, it does seem like having an option for manage_externals to checkout the branch properly rather than as a detached head would be useful here.

[ekluzek]

The way I recommend is that if the component in question is at the top level (in Externals.cfg) you edit it there and use manage_externals to manage the branch. If you have to point to something that's in a lower level -- right now you have to do it with git commands.

One advantage of using manage_externals for Externals.cfg is that the CIME case system documents manage_externals results along the way. So if the top level Externals.cfg file is correct it's going to add it to the documentation of the case or test. It'll also give good documentation when you run the status command. That's part of why I think it's good when you can to use manage_externals for at least the top level.

[ekluzek]

@cacraigucar also has a good point about when you have uncommitted mods. Sometimes you have something small that you don't really want to commit or create a branch for. One answer is git's "stash" command that allows you to keep a few of these types of things around with a little less overhead. It still has some overhead though.

Personally I usually work around it either by using "git stash" or by saving a local copy of what I changed before running manage_externals and then copy it back in afterwards. Saving a copy works when it's a single file, which is my typical case for wanting to do that.

[ekluzek]

@Katetc I'm not sure what you mean by this...

I think it would be very valuable to have manage_externals ignore the changes to an Externals.cfg file, so that all checkouts can be done using one method and one tool.

So are you saying it would use the latest version of Externals.cfg in git and ignore the modified version? That would be a problem while you are developing a new branch or tag or even a new git repository for that matter.

I guess I could see an option where manage_externals was told to abort if Externals.cfg has been modified. I think maybe you are thinking of the case where you've checked out a given tag, and really want the externals to match exactly what that tag was? If so that does make sense in those cases that I've cloned something that I just want to point to a given very specific tag, and don't want it to ever change from that (like for control simulations).

[Katetc]

@ekluzek , sorry I was a bit vague there. I meant that manage_externals should ignore changes to Externals files in subdirectories. Such as the Externals_CISM.cfg file. Currently, if that one is changed, then manage_externals will not proceed due to the modification. I think it should treat a subdirectory where only an Externals.cfg file is changed as being unchanged. In other words, treat Externals.cfg files in sub directories the same way it does the ones in the main directory.

[billsacks]

Thank you all for your responses so far. For now I just want to reply to @Katetc 's latest comment:

manage_externals should ignore changes to Externals files in subdirectories. Such as the Externals_CISM.cfg file. Currently, if that one is changed, then manage_externals will not proceed due to the modification. I think it should treat a subdirectory where only an Externals.cfg file is changed as being unchanged.

This is a lot easier said than done, for two reasons (and I acknowledge that I'm the one who brought this up based on earlier conversations with @ekluzek , so I'm not pointing a finger at you for asking for it here):

1. Currently, manage_externals just checks the git status of each external; if there are modified files, it calls the tree dirty:

https://github.com/ESMCI/manage_externals/blob/c33a3bd2a856dec33febf7f3fda30a4b0b9af608/manic/repository_git.py#L572-L597

What you're asking for would require parsing the output and ignoring files that match a certain pattern. Note that there is no required naming convention of files like Externals_CLM.cfg - that's just a convention. So this gets even trickier to do robustly.

You said:

In other words, treat Externals.cfg files in sub directories the same way it does the ones in the main directory.

but it's not that easy. Externals.cfg in the main directory is ignored implicitly, because it isn't a member of any externals.

2. Assuming we could take care of (1), then there is the issue of what happens if if manage_externals tries to update an external that has a modified Externals_FOO.xml file. In some cases, this would lead to problems in the update, which is what we were trying hard to avoid. For this reason, I feel that ignoring some unmodified files is ill-advised.

Sorry, I forgot about point (2) when suggesting this possibility in my original comment. I am now thinking we really shouldn't do this.

Now what we could possibly do is generally relax the checking in manage_externals, so that it still allows an update in the case of a dirty sandbox, as long as it isn't trying to update a component that is itself dirty. (So if you have modified components/cism/Externals_CISM.cfg, then manage_externals will run happily as long as the top-level Externals.cfg isn't trying to change components/cism.)

But I'm also having trouble reconciling these thoughts with your earlier comment:

I really, really prefer to work with the Externals.cfg files. I find that they are an important way to document exactly which versions of which components you are using for your experiments. This is super important for papers and reproducibility.

For this to work well, don't you need to have your changes committed to a branch, and have that branch pushed to somewhere on GitHub?

[ekluzek]

Note, I'm setting up a time to discuss this tomorrow at 11:00am MDT. Ping me if you want to be added to the group.

[Katetc]

Ok, so I see some of the issues here. Thanks for explaining that. There is a lot to consider here and I'm having a hard time framing my thoughts in a coherent message this afternoon. I agree with everything you said Bill, but I find it hard to believe that the ONLY solution to this problem (ie, in my sandbox, I want to run with a different src_cism than the release) is that we need a new wrapper branch every time, or stop using manage_externals (and then your externals don't match the Externals_FOO.cfg file).

Though, I wrote another entire comment about how checking your code into a branch is generally a good idea, and maybe we should just suggest branches off people's forks to address this. But, that could be tough for plenty of other reasons. So, I deleted my comment.

We can talk more about this at the meeting tomorrow.

[mnlevy1981]

There is a lot to consider here and I'm having a hard time framing my thoughts in a coherent message this afternoon

Me too, but @ekluzek I'd like to be included in the discussion tomorrow (hopefully I'll have some clearer thoughts by then)

In my mind, it would be great to include the features @Katetc and @cacraigucar are requesting:

1. A way to signal that to manage_externals that some files can differ from what's currently in git without aborting the entire checkout
2. A way to specify that you want to end up on the head of the branch instead of in a detached head state

So far I've had a couple of awesome idea that were decidedly less awesome by the time I got halfway through writing them up, but maybe others can expand on my half-baked ideas tomorrow :) (Basically, I'd like a .manage_externals_ignore file that signals to manage_externals that it's okay if a specific file has been modified, but I think it maybe only works in the very narrow scope where the current checkout / detached head matches where manage_externals wants it to be? That avoids the pitfall of pattern-matching @billsacks mentioned in #139 (comment), while also matching his second criteria for when to allow "dirty" checkouts... but with the added safety feature that only certain files are allowed to be modified)

[billsacks]

1. A way to signal that to manage_externals that some files can differ from what's currently in git without aborting the entire checkout

I just remembered that I already opened this issue: ESMCI/manage_externals#112 . If people feel that's a good idea, then it would take care of this point, at least in some use cases.

A way to specify that you want to end up on the head of the branch instead of in a detached head state

We (largely you, @mnlevy1981 , together with @gold2718 ) sketched out a design for this in ESMCI/manage_externals#34. I don't feel this needs more discussion right now.

What both of these issues have lacked is someone with the time to implement them.