When routing fails, exit code should not be `0`

[kasra-keshavarz]

Specifically this part of the routing code:

MESH-Dev/Routing_Model/RPN_watroute/sa_mesh_process/rte_module.f90

Lines 923 to 931 in 538766d

	if (exitstatus /= 0) then
	if (dtmin <= mindtmin) then
	write(*, '(a15, a43, f6.1)') 'route yields a ', 'negative store2 value at dtmin = mindtmin: ', mindtmin
	write(*, '(a25, a50)') 'It''s likely that qo1 is ', 'so large that store2 is negative even with qo2=0.0'
	write(*, '(a29, a50, a50)') &
	'If this run was started from ', 'shed2flowinit utility, then try lowering the QI, ', &
	'QO, and STORE ratios until this error is resolved'
	write(*, *) 'Else rerun with a smaller value of dtmin'
	stop

My suggestion is to change stop to stop exitstatus (uncertain about the Fortran syntax).

Technically, that if clause is executed whenever the exitstatus variable is non-zero.

This helps tracking the successful execution of the model. Current exit code (of 0) suggests model has been executed successfully, in *nix systems at least.

Also, a similar code exists here:

MESH-Dev/Routing_Model/RPN_watroute/code/rte_sub.f

Lines 1075 to 1086 in 538766d

	if (exitstatus .ne. 0) then
	if (dtmin .le. mindtmin) then
	write(*,'(a15,a43,f6.1)') 'route yields a ',
	* 'negative store2 value at dtmin = mindtmin: ',
	* mindtmin
	write(*,'(a25,a50)') 'It''s likely that qo1 is ',
	* 'so large that store2 is negative even with qo2=0.0'
	write(*,'(a29,a50,a50)')'If this run was started from ',
	* 'shed2flowinit utility, then try lowering the QI, ',
	* 'QO, and STORE ratios until this error is resolved'
	write(*,*) 'Else rerun with a smaller value of dtmin'
	stop

I assume the latter is not executed and is legacy WATROUTE/WATFLOOD code.

[kasra-keshavarz]

I did a quick syntax search; I think to erroneously STOP the Fortran program, one should use ERROR STOP exitstatus? Not sure. Just a suggestion, as it makes it easier to track when a program exists successfully or not.

[dprincz]

That code is actually newer, added by ECCC to support its operational systems. However, rte_sub is also wrapped in a condition enforcing dtmin = max(mindtmin, dtmin) that appears in the MESH connector in rte_module so it won't trigger from MESH (this doesn't exist running standalone WATROUTE/RTE, i.e., rte.f).

ERROR STOP is a newer syntax (F2008). I'm not sure how universally supported it is by the compilers we test.

The legacy messages in RTE usually use the syntax STOP [scalar-char-constant]. A lesser invasive change could be to amend the newer stop statements to follow a similar approach. -- However, I'm not sure how those program returns are interpreted, unless just as a non-zero exit and if that's sufficient for your purposes.

I should mention WATROUTE/RTE in a standalone mode signals a "normal" exit like this, and I'm not sure how common this might come up in other programs:

stop 'watroute: normal ending'

The approach could erroneously tell your interpreter the program did not exit normally. -- Something to consider.

[dprincz]

To simplify the suggestion, would this be specific for a MESH wrapper related to the calibration workflow work, or part of a model-agnostic interpreter? Is there a specific interpretation the workflow is looking for? -- In which case, we could adapt the model.

[kasra-keshavarz]

So, on *nix systems, I capture the exit code of programs with a

$ echo $?

Based on Advanced Bash-Scripting Guide, anything non-zero is erroneous termination.

I did a quick test with Fortran, on Linux/Mac and tested the STOP statement with different values and tried to see what the exit code is.

Here are the simple programs:
test_none.f90:

program exit_code_example
  implicit none
  print *, "This program will exit with no status code."
  stop
end program exit_code_example

test_1.f90:

program exit_code_example
  implicit none
  print *, "This program will exit with status code 1."
  stop 1
end program exit_code_example

test_0.f90:

program exit_code_example
  implicit none
  print *, "This program will exit with status code 0."
  stop 0
end program exit_code_example

And here are the executions:

$ ./test_0
 This program will exit with status code 0.
STOP 0
$ echo $?
0

and

$ ./test_1 
 This program will exit with status code 1.
STOP 1
$ echo $?
1

and

$ ./test_none 
 This program will exit with no status code.
$ echo $?
0

I think on Windows, one can capture the exit code with echo %errorlevel%. I tried to respect this in datatool (an example), etc. as I know HPC schedulers (including SLURM) consider the exit code of a process as the success/fail status of a job. In cases where defining dependencies based on a successful execution of a program on HPC environments is needed, this can cause misinterpretation.

[kasra-keshavarz]

To simplify the suggestion, would this be specific for a MESH wrapper related to the calibration workflow work, or part of a model-agnostic interpreter? Is there a specific interpretation the workflow is looking for? -- In which case, we could adapt the model.

Not necessarily needed for calibration, though it helps. For calibration, I have set the evaluation to fail whenever a score cannot be calculated (e.g., model is not properly run, the outputs are weird, the model crashed, etc.)

But I think it would generally help tracking the exit status of MESH.