Flux: Shut Down 2min to Walltime

Author: ax3l

Docs/source/install/hpc/tuolumne.rst

@@ -271,6 +271,9 @@ The batch script below can be used to run a WarpX simulation on 1 node with 4 AP
 Replace descriptions between chevrons ``<>`` by relevant values, for instance ``<input file>`` could be ``plasma_mirror_inputs``.
 WarpX runs with one MPI rank per GPU and uses 21 (of 24) CPU cores (3 are reserved for the system).
 
+The batch script below also :ref:`sends WarpX a signal <running-cpp-parameters-signal>` when the simulations gets close to the walltime of the job, to shut down cleanly.
+Adjust the ``FLUX_WT_SIG`` and ``WARPX_WT`` to modify or disable this behavior as needed.
+
 .. tab-set::
 
    .. tab-item:: GPU

Docs/source/usage/parameters.rst

@@ -354,6 +354,8 @@ Overall simulation parameters
     If set, the environment variable ``OMP_NUM_THREADS`` takes precedence over ``system`` and ``nosmt``, but not over integer numbers set in this option.
 
 
+.. _running-cpp-parameters-signal:
+
 Signal Handling
 ^^^^^^^^^^^^^^^
 

Tools/machines/tuolumne-llnl/tuolumne_cpu.flux

@@ -26,6 +26,11 @@
 EXE="./warpx.2d"
 INPUTS="./inputs_hist_10.input"
 
+# clean shutdown close to walltime (or checkpoint)
+# https://warpx.readthedocs.io/en/latest/usage/parameters.html#signal-handling
+FLUX_WT_SIG="--signal=SIGUSR1@120s"
+WARPX_WT="warpx.break_signals=USR1"
+
 # enviroment setup
 if [[ -z "${MY_PROFILE}" ]]; then
     echo "WARNING: FORGOT TO"
@@ -49,7 +54,7 @@ export OMP_NUM_THREADS=21
 
 # start MPI parallel processes
 NNODES=$(flux resource list -s up -no {nnodes})
-flux run --exclusive --nodes=${NNODES} \
-  --tasks-per-node=4 \
-  ${EXE} ${INPUTS} \
+flux run ${FLUX_WT_SIG} --exclusive --nodes=${NNODES} \
+  --tasks-per-node=4           \
+  ${EXE} ${INPUTS} ${WARPX_WT} \
   > output.txt

Tools/machines/tuolumne-llnl/tuolumne_mi300a.flux

@@ -26,6 +26,11 @@
 EXE="./warpx.2d"
 INPUTS="./inputs_hist_10.input"
 
+# clean shutdown close to walltime (or checkpoint)
+# https://warpx.readthedocs.io/en/latest/usage/parameters.html#signal-handling
+FLUX_WT_SIG="--signal=SIGUSR1@120s"
+WARPX_WT="warpx.break_signals=USR1"
+
 # enviroment setup
 if [[ -z "${MY_PROFILE}" ]]; then
     echo "WARNING: FORGOT TO"
@@ -52,8 +57,8 @@ GPU_AWARE_MPI="amrex.use_gpu_aware_mpi=1"
 
 # start MPI parallel processes
 NNODES=$(flux resource list -s up -no {nnodes})
-flux run --exclusive --nodes=${NNODES} \
+flux run ${FLUX_WT_SIG} --exclusive --nodes=${NNODES} \
   --tasks-per-node=4 \
   ${EXE} ${INPUTS} \
-  ${GPU_AWARE_MPI} \
+  ${GPU_AWARE_MPI} ${WARPX_WT} \
   > output.txt