set SCR_END_TIME in srun scr_run script

[rhaas80]

SCR's srun based scr_run script (in scripts/TLCC/scr_run.in) does not set SCR_END_TIME thus the only way scr_env_seconds_remaining can obtain them is if libyogrt (https://github.com/LLNL/libyogrt) is present such that

  #ifdef HAVE_LIBYOGRT
    secs = yogrt_remaining();
    if (secs < 0) {
      secs = 0;
    }
  #else

can execute.

Since yogrt is listed as optional in README.md it may be nice if scr_run provided a fallback in case it is not installed.

Something like:

export SCR_END_TIME=$(date -d $(scontrol --oneliner show job $SLURM_JOBID |  perl -n -e 'm/EndTime=(\S*)/ and print $1') +%s)

[adammoody]

Great idea!

Thinking through which should take precedence if both are available...

On the one hand, yogrt can react to dynamic changes in the end time, for example if the user or admin changed the runtime of the job after it had already started. That could be somewhat common in the case that admins are clearing running jobs from a machine to make room for high priority jobs, like dedicated application testing in LLNL-speak.

On the other hand, maybe the user intentionally set SCR_END_TIME to something less than the job's runtime, in which case they really want SCR_END_TIME to be used. However, to exit out early, they can still set SCR_HALT_SECONDS, which is more appropriate anyway.

So if SCR_END_TIME is defined and libyogrt is available, we should use libyogrt.

[rhaas80]

Right now YOGRT takes precedence due to the #ifdef so that seems fine. While using (outside of scr proper so it is a hack admittedly) scr_env_seconds_remaining in my test code I also noticed that, at least on Catalyst, only on rank 0 (or possible only on the node hosting rank 0, I only have 1 rank per node for each of my 4 nodes in my test) did I receive non-zero remaining values from scr_env_seconds_remaining.

[adammoody]

Oh, right. Good point about the #ifdef.

For the non-zero only on rank 0, is that with libyogrt? I think libyogrt is only meant to be called from one rank.

https://github.com/LLNL/libyogrt/blob/75f3f448f5e0e3aeaffe4767127f0187e570831c/src/yogrt.h#L33

The SCR code should only be calling it from rank 0 and then bcasting the result to other ranks.

scr/src/scr.c

Line 186 in 4074ee5

long int remaining = scr_env_seconds_remaining();

In hindsight, it might have been nice to make that clear in the comments or protect against it somehow in the implementation. Anyway, now you know.

[adammoody]

To add to that a bit, if all processes were to make a call to get the time remaining, it's important that they all get a consistent result. For example, if each process in the job could actually call libyogrt, you probably would still want just one process to make the call and bcast its value. Otherwise different procs could end up with different values, and then some might decide it's time to exit and others would not leading to hang.

[rhaas80]

Pull request ready for review for SLURM is

#266

(really a one line change).

For PMIX things are currently broken in various way. This pull request (mostly a WIP asking for comments) querys PMIx for the remaining time using the interface described in https://pmix.github.io/standard/RFC/query-time-remaining-in-allocation :

#267

[rhaas80]

For SLURM added in 102788c. PMIX still not in but likely much less pressing since PMIX is currently broken anyway so cannot be possibly in use anywhere.