Monitoring latency

[pierre-yves-monnet]

On the monitoring page, add the latency information.
For a worker execution:

• monitor the start time of the execution, and the end time (BeginWorker, EndWorker)
  At the end of the execution, the execution is registered in a queue somewhere, to be addressed by a different thread (just an idea of how to address that)
  Then a call can be sent to the server, to get the information "when the task was available" and "when the task was executed". (BeginTask, EndTask)

With this information, we can:

• calculate the total latency: EndTask-BeginTask - (EndWorker-BeginWorker). This time is the total latency for the system
• calculate the start latency: BeginWorker-BeginTask. This information is the time a worker needs to be involved when the task is ready. This information is not very safe: it implied the two machines are perfectly synchronized on their clock.
  But if this delay is big ( > 500ms), this is maybe a good indicator to know that we can improve the number of worker (or to decrease them according the result of the POC).

Idea is to display a graph with these two pieces of information on the last 24 hours and to get an average.

Additional question: how can we save this information? If we stop/start the application, we don't want to lose it.