[Slurm] Alternatives for job completion monitoring

[sylvlecl]

• Do you want to request a feature or report a bug?

Feature

• What is the current behavior?

In order to monitor the completion of jobs submitted to Slurm, we use files and filesystem polling.
Depending on the polling frequency, this introduces some performance cost (delay between the end of the task and the time when the computation manager identifies it as completed), and some load on the underlying filesystem, in particular when multiple processes using a computation manager are running.

• What is the expected behavior?

We could be able to configure the way the completion monitoring is performed.
Polling will be one implementation of this functionality.

Other interesting implementations would be :

1. A very simple in house networking protocol, for example implemented with netty.
2. Using a message broker (kafka, rabbitmq ...) : this should probably be left for implementation by client projects

• What is the motivation / use case for changing the behavior?

Improving perceived performances while relieving the filesystem.

• Please tell us about your environment:
  • powsybl-hpc version: 2.7.0

[yichen88]

If slurm is in local mode, we can simply register a WatchService on flagDir.

[sylvlecl]

Yes, but the problem is that even in "local" mode, there are good chances that the flag dir is actually on a shared filesystem, for instance a nfs mount, so that slurm nodes can access it. In that case, the watch service will probably not work (or be implemented with polling).