RFC: How should Legion pin Realm versions?

[elliottslaughter]

As we approach the Realm split, Legion will need to download and build specific Realm versions. The question here is how to do that pinning.

Below I outline four approaches I'm aware of, but this is not an endorsement of any specific approach.

1. CPM (CMake Package Manager)

   We already use CPM to build Realm inside the Legion CMake build, and CPM supports the ability to pin a specific commit ID. However this would not address the Makefile, so we'd need a separate solution there.

   CPM can (probably?) work with caching so this approach is CI-friendly.

   One disadvantage of this (and all build-side tooling) is that users can't download the Realm code without actually running the build tool. One has to run CMake to get CPM to download the Realm source, and if we build something similar for the Makefile it would be the same thing.

2. Git submodules

   We could make Realm a Git submodule. This would mean that users would have to start cloning Legion with git clone --recursive (or then subsequently run git submodule update --init).

   One advantage of submodules is that users can obtain the corresponding Realm source without running CMake or any other build tool.

   With submodules there is also the question of where the submodule lives. You can use an absolute (https://github.com/StanfordLegion/realm.git) or relative (../realm.git) link.

   I don't see how you'd cache a submodule, so we might hit GitHub rate limits, unless we use a relative link and mirror Realm into Gitlab (but that adds complexity).

3. Git subtress

   git subtree is a contrib module included in Git for many years now. It's basically like git submodule except you literally copy the code into the target repository. There are a variety of modes and flags you can use; maintaining history is optional.

   Subtrees mainly benefit users by removing the second step to download code. It's just there, directly in the repo. On the other hand it takes some discipline to use it correctly (e.g., one would want to treat the realm/ subdirectory as read-only from inside the Legion repository).

4. Custom tooling

   This could be practically anything, but one possibility would be to store a file named REALM_VERSION (name subject to change) that contains the commit ID of a specific Realm version. Then various tools (CPM/Makefile/etc.) can refer to that file to download the corresponding Realm version.

   I'm sure that CPM can be made to download a commit ID from a file (CMake can read files), I'm not sure what we'd do in the Makefile.

   If we wanted to enable users to download the Realm code without running one of the build tools (CMake/Make) then we'd need a custom script for it.

There may be other dimensions or options I haven't thought of. Please respond with any suggestions/questions/concerns below.

[jpietarilagraham]

#1 CPM. We only use cmake and almost always build legion through spack anyhow.

[muraj]

The main concern I have is being able to trigger, remotely from github, the legion-ci with a particular commit from github, at least for the time being while realm's CI is ramping up coverage. I don't really care what way legion wants to go so long as I can at least satisfy that one requirement.

For option 3, git subtrees, IIRC, are more meant as a way to have multiple checkouts of the same repo for different branches / references all tied to the same git checkout of the entire repository, allowing for a more SVN/CVS like workflow. I'm not sure I can fully envision how that plays into the issue at hand.

For option 1:

CPM can (probably?) work with caching so this approach is CI-friendly.

Yes, we cache all the source files locally in CI today. But yes, you're right, there isn't really anything similar for the Makefile system unless you wanted to have it run a script or something that would download, build, and install Realm from some commit hash yourself.

Overall, I'm in favor of option 2 as it satisfies all the current requirements and has the least impact for everyone. That said, forcing a relative URL in option 2 and mirroring the repository in gitlab I believe wouldn't allow for remote legion-ci workflow triggering since the gitlab mirror would be out of sync and not have the commit we want to test.

For the caching issue -- @elliottslaughter we should be able to work around that by tooling the CI to look at the commit hash for the submodule and manually checking it out within the CI script. Then we just cache the full source, which can be keyed off that commit hash so it's reusable across jobs, and we'll just build it separately like we do today in CI. That's basically what we do for gasnet and ucx anyway, the question is more where we get the HASH from, and I would say, if it's not in a CI variable(s), then by default pull it from the submodule without having to checkout the submodule. The CI variable(s) would allow us to override the commit SHA and pull it from where ever.

I know people have a certain distaste for submodules, but I haven't really heard of any concrete reason why they're bad as of yet.

[mpokorny]

I have no strong opinion, but I want to comment on my current workflow for awareness. I only use the Legion CMake build system, not Makefile, so I won't comment on the Makefile aspects of the question.

There's one particular usage pattern for building Legion that I am using a lot lately, which comes down to building with my modifications for Kokkos support in Realm. These modifications exist in multiple forms: as a branch in a local repo, as an MR branch in the Legion gitlab repo, and as a patch file that works with a modified Spack package (which I share with others on my team). The use of patch files would require downloading the Realm source prior to running cmake, but I could probably live without that option if a user-provided Realm source repo and commit id could be used by the build system.

When I work on my Realm code modifications, I always build Realm via a Legion build because the only test code that I have is Legion code. In this case, it's convenient to have a local copy of Realm on which I can work, without needing to go through an intermediate commit to some other repo prior to every Legion build.

I'm not familiar enough with CPM to comment on it. I think that using a git submodule for Realm would allow me to continue working as I have in the past.

[syamajala]

My original vote was for subtree but I have no idea how that complicates CI and stuff. It just seems like the most transparent thing to legion end users, since things would basically look the same as they do today.

[mariodirenzo]

My two cents on this is that I have been using the submodule approach for HTR (with Makefile) and HTR++ (with CMake), and I did not encounter any specific issues. Though HTR has a much simpler repository, I find the ability to easily modify and recompile one of the submodules very useful during debugging.

[muraj]

Linking to an issue where Mike had a write up about this that I recently ran across:

StanfordLegion/realm#138 (comment)

[elliottslaughter]

Documenting my own opinion now that most people I've asked have had a chance to comment.

I'm ok with submodules (option 2). There are some annoyances but I think they're minor, or at least less significant than the other options.

But it's still worth documenting the pitfalls so we're all aware. The main ones I can recall right now are:

1. Users forgetting to clone with --recursive and/or not realizing they need to run git submodule update --init. Any experienced Git user will know this, but we have enough novice Git users that someone will inevitably trip up on this. Not hard to correct but it'll result in some noise on the issue tracker.
2. The first time you introduce a submodule, if it exists at a path that previously held actual files, Git will refuse to initialize the submodule if anything in that directory is dirty (including ignored files that will not show up in git status). This is just something to be aware of, so when we do the switch it might be best to do a fresh clone to avoid gotchas.
3. With submodules, it becomes incredibly important to never overwrite commits on the upstream repo. The downstream repo only maintains a commit hash, so if anyone force-pushes the upstream repo, that hash becomes invalid. While I'd expect this to be a non-issue for Realm master, if we ever have a Realm branch that we want to coordinate with Legion, we need to be sure to not commit any Legion-side changes to the submodules until the Realm branch has merged. Otherwise while bisecting you could hit Realm commits in Legion that actually do not exist.
4. I can't remember the specifics, but submodules are the only feature of Git that have ever caused someone I know to get into a state where I couldn't help them, and couldn't figure out how they'd gotten into that state. Almost everything else you can do in Git is reversible if you look at the reflog, but the combination of submodules + modifying the contents of submodules seems to be quite tricky. This was a long time ago so perhaps this is fixed in modern Git, but just wanted to mention it.

For the caching issue

Actually I realized after posting this that the Git commit IDs for submodules live in the root .git object store, which means that if we use a git clean -fxd approach they should be preserved across CI jobs. So I think this may "just work" without any further intervention. But certainly worth investigating.

[lightsighter]

I think I would prefer subtrees (without Realm's commit history). My justification for this is as follows:

1. As @syamajala points out, this provides backwards compatibility for workflows for ourselves and our users.
2. I'm still very scarred from our use of submodules with rdir. As described by @elliottslaughter, I too encountered several different situations with submodules where the only solution was to delete the whole repo and check it out again. Doing merges from branches which were aligned to different commits of submodules was really painful. Any kind of merge conflicts were nearly impossible to resolve correctly.
3. Most of existing tooling (including the Make build system) will just continue to work as-is without any changes.

The way I look at it, really submodules and subtrees are the only viable options and the choice between them comes down to how frequently we plan to be updating the Realm commit. If we think we're going to be frequently updating then we kind of have to go with submodules because they are easier to bump the Realm commit. Of course this comes with a bunch of crap that you have to deal with outside of bumping the commit and we don't get as nice of a workflow for the rest of everyday processes. However, if we're going to bump the commit infrequently, then subtrees are the better option since it maintains all of our existing workflows in a seamless way for both developers and users, and we only have to do manual stuff infrequently to update the Realm commit.

I want it to be the case that Legion will infrequently need to bump the Realm commit. In fact, I would prefer that the workflow be that we do a Legion release, and then after the release, we will attempt to increment Legion's Realm commit to the next most stable Realm commit so that we can then run with that for the next quarter. If that is the case, we're only talking about bumping the Realm commit four times per year. If people can make a compelling case for bumping the Realm commit more frequently then that, then submodules start to come back into play.

[elliottslaughter]

Summarizing discussion from meeting today:

• We will tentatively try out git subtrees with infrequent Realm bumps. If that doesn't work (e.g., bug fixes require frequent updates, and/or subtree process is too cumbersome), we'll switch to git submodules.
• We will aim for infrequent releases to begin with unless that is shown not to work for whatever reason.

Process from here:

• Resolve any remaining outstanding build issues.
• Freeze /realm directory inside of Legion. This means we will no longer take Realm MRs in Legion Gitlab.
• Sync Legion's Gitlab with Realm's GitHub. Ensure both are equivalent.
• Issue a release candidate for Realm via Realm GitHub.
• Continue Realm development as normal in GitHub from here.

[lightsighter]

Importantly, we should establish some ground rules for the Realm sub-tree:

• We'll be squashing all Realm commits when using Realm sub-trees so that the Realm commit history doesn't pollute the Legion history.
• The Realm sub-tree should be treated as read-only, meaning we won't be attempting to push back to the Realm sub-tree from the Legion tree. This should eliminate most of the major issues with sub-trees.