What should the default for CHPL_LAUNCHER be when CHPL_COMM=none on linux64?

[jabraham17]

Recently, I merged #26236 which changed the defaults for CHPL_LAUNCHER.

Prior to that PR, that logic was

1. if we are on a cray-x* or hpe-cray-*, use a slurm/aprun launcher as available
2. If COMM=gasnet, use a gasnet launcher

• if we are on cray-cs or hpe-apollo and slurm is available, use a slurm gasnet launcher

3. otherwise, if we are on cray-cs or hpe-apollo, use slurm as available

See #26170 for a more indepth discussion of the history here, but essentially these defaults arose simply because it was the easiest thing to do for our testing infrastructure. Based on the fact that users on a slurm system probably want to be using slurm (and not accidentally running on a login node), #26236 changed the default to the following

1. if we are on a cray-x* or hpe-cray-*, use a slurm/aprun launcher as available
2. If COMM=gasnet, use a gasnet launcher

• if slurm is available, use a slurm gasnet launcher

3. otherwise, use slurm as available

This meant any generic system that also had slurm installed would default to a slurm launcher, regardless of the value of CHPL_COMM. This was for a number of reasons

• The previous behavior of defaulting to LAUNCHER=none seems to have come from a desire to make our test infrastructure easier, not about whats best for users.
• If a user is running on a system with slurm installed, even with COMM=none, its more likely they are looking to run Chapel code on the compute nodes using slurm.
  • e.g., they are looking to debug chapel code with COMM=none on a compute node
  • e.g., they really only need a single node and want to compile with COMM=none to reduce the generated code that comes from a multi-locale build
  • e.g., they are running GPU code on a cluster and don't need/want multinode. And more than likely, there are no GPUs on the login node
• Why would a user want to run on a slurm cluster and not use a slurm launcher? The only thing that comes to mind for me is a large system with long queue wait times. But if thats the case, the login node is likely a bad place to run large computations.

This didn't feel like a big change, as on HPE/Cray systems with slurm the default LAUNCHER for COMM=none was already slurm. All this changed was removing the platform specialization. However, after making these changes there was a feeling that this was a step too far. This issue is about the specific case of defaulting to CHPL_LAUNCHER=slurm-srun when CHPL_COMM=none, slurm is available, and we are on a generic clsuter (i.e. not a specific HPE/Cray system)

[bradcray]

In reading #26170 and hearing about #26236, the cases I recall hearing about are the ones on the issue and the changes to behavior when CHPL_COMM!=none, which sounded good to me. I asked Jade to open this issue because I felt as though the change to CHPL_COMM=none configurations was sufficiently significant and surprising that it warranted more discussion than (I'm aware) it's received (e.g., I didn't know about it today despite reading the issue, talking with Jade about the PR, and reading the PR's merge message).

The part that gives me pause in the rationale given above is:

If a user is running on a system with slurm installed, even with COMM=none, its more likely they are looking to run Chapel code on the compute nodes using slurm.

Specifically, it doesn't seem obvious to me that this is necessarily more likely than running on the current node. If CHPL_HOST/TARGET_PLATFORM is inferred to be linux64 and CHPL_COMM is none, I don't consider it obvious whether:

• the user is running on a system in which a linux64 node plays the role of a login node and they want to launch to the "compute" nodes
• the user is running on a system in which the login and compute nodes are interchangeable, such that running on the node that they're currently on would be reasonable or possibly even what they want (maybe they're already within a slurm allocation?)
• the user is running on a desktop system that just happens to have srun/salloc installed

We could potentially attempt to distinguish these cases by heuristically beefing up the choice of default CHPL_LAUNCHER to check whether we're already within a slurm partition, see whether the slurm launcher logic works, or things like that, but to me that feels like too much complexity / insufficiently K.I.S.S. (while still likely being imperfect—so more complexity in implementation and documentation/understanding without being an obvious win).

[edit: chapcs is a good example of the second bullet above. It used to be that if I wanted to study the behavior of a comm=none program or its performance, I'd log into a node, potentially reserving it through slurm if I was doing an intensive study, and run it. But now, the default is to run it on a node other than the one I'm logged into:

chapcs11 (main)$ ./hello6-taskpar-dist -nl 1
Hello, world! (from locale 0 of 1 named chapcs04)

]

I also think that some of the examples above:

e.g., they are looking to debug chapel code with COMM=none on a compute node
e.g., they are running GPU code on a cluster and don't need/want multinode. And more than likely, there are no GPUs on the login node

imply that there's some sort of asymmetry between the login node and compute nodes that suggests that user may need to set other things in order to cross-compile for the compute node anyway (e.g., presumably I'd have to set at least CHPL_GPU in the second case?). So, given the amount of care that cross-compiling or cross-debugging tends to require anyway, it doesn't seem particularly onerous to have users with such workflows to also set CHPL_LAUNCHER to say "despite the fact that I'm only asking to run on one node, it isn't this node that I'm currently on."

In addition, historically, I can't think of any cases where the current behavior on linux64/comm=none/launcher=none systems has been surprising, confusing, or troubling to users, yet I can imagine the new default being surprising. Say I'm new to Chapel and maybe distributed-memory programming in general, I've tried it on ATO or codespaces, and have now downloaded it and am trying things out. To have it suddenly generating two binaries rather than one and interacting with slurm, possibly in ways that won't work (maybe I'd need to specify a queue, an allocation, or some other policy to make the slurm launcher work correctly as often seems to be the case?) feels sufficiently different from what I'd get or be used to from most languages that it feels like a potential source of confusion or churn to the novice user. I'm thinking specifically of all the "How do I pipe additional arguments through to slurm via the Chapel launcher?" questions we seem to get from people who are knowledgeable about slurm, and imagining new-to-slurm users hitting similar issues on their first run. Meanwhile, I don't think asking an expert user who wants to do one of the things above to set CHPL_LAUNCHER is asking too much.

A lot of my argument boils down to this: Most languages compile to a single binary that runs on the node you're on by default. This seems like a reasonable default for Chapel as well, at least until the user starts wanting to do something different (using multiple nodes by setting CHPL_COMM, cross-compiling, interacting with slurm, etc. by setting other things).

[mppf]

FWIW, we could also consider requiring CHPL_LAUNCHER be set explicitly in the setting of CHPL_COMM=none and slurm found, but not on a Cray xc/ex/etc.

Another potential situation is that a user might install slurm commands on their PC (either to play with locally, or through confusion). E.g. Ubuntu has a slurm-wlm package. I'd generally view it as a bug on our part if Chapel breaks just because an Ubuntu package is installed (where it would work without that package installed). I am worried enough about this scenario that I think we should adjust something here.

[bradcray]

FWIW, we could also consider requiring CHPL_LAUNCHER be set explicitly in the setting of CHPL_COMM=none and slurm found

For me, that'd be better than the status quo since it would reduce the chances of unexpected surprises, but my preference would still be to have it default to none in order to reduce the number of steps required to get an initial desktop install of Chapel working and behaving similarly to most languages (that is, by working under the assumption that running the program locally is the least surprising default behavior).

By extension, if I was working on a system that happened to have both slurm and PBS installed, I'd prefer to have CHPL_COMM=none default to CHPL_LAUNCHER=none rather than being prompted to choose one of the launchers. In contrast, for a CHPL_COMM!=none run on such a system, getting a prompt to select a CHPL_LAUNCHER feels very reasonable and helpful.

[bradcray]

I got bitten by an unanticipated consequence of this change this afternoon when my paratest that usually runs in less than an hour was only through half the directories 3 hours later. I believe this is because I had not set CHPL_LAUNCHER=none in my .bashrc or equivalent, s.t. each program launch is now going through slurm. Granted, I could make that change to my .bashrc now that I've realized it, but it feels like another example of the typical thing I'd expect from a programming language (execute a single-locale binary where I am rather than elsewhere) being more typical / less surprising,

Over dishes, I also came up with one more variation where "behaving like a typical programming language by default" seems more attractive to me, which is that I'm building a Chapel CHPL_COMM=none executable with no intention of running it on the system where I happen to be compiling it. Maybe I'm going to ship it off to you in binary form, or maybe I'm using this system to cross-compile for some other platform or chip. In such cases, it seems as though the presence or absence of slurm on the system where I'm doing the build shouldn't affect the application's default, and would be less surprising if it matched the typical behavior when compiling a single-node binary in C++, Fortran, etc.

[jabraham17]

Most languages compile to a single binary that runs on the node you're on by default. This seems like a reasonable default for Chapel as well, at least until the user starts wanting to do something different (using multiple nodes by setting CHPL_COMM, cross-compiling, interacting with slurm, etc. by setting other things).

I've been thinking about this a lot this week and I keep coming back to this point, which resonates with me a lot. If I am running locally and not on a known cluster system, the least surprising thing we can do is use CHPL_LAUNCHER=none.

I got bitten by an unanticipated consequence of this change this afternoon when my paratest that usually runs in less than an hour was only through half the directories 3 hours later. I believe this is because I had not set CHPL_LAUNCHER=none in my .bashrc or equivalent, s.t. each program launch is now going through slurm. Granted, I could make that change to my .bashrc now that I've realized it, but it feels like another example of the typical thing I'd expect from a programming language (execute a single-locale binary where I am rather than elsewhere) being more typical / less surprising,

I want to push back on making a decision here about whats best/easiest for our paratest tool. To my mind, that is optimizing for the wrong case. We should pick a good default based on whats best for typical usecases, I don't consider our paratest tool a typical use case.