State of RE Worker and Manager has confusing edge condition

[whs92]

Edge Condition in Queueserver Status

The queuserver can report three different states in a request for status

• re_state. The RE state. The state of the run engine (idle, running, paused)
• worker_environment_state
• manager state

If I client wants to answer the question: "Is the stopped and not going to do anything else now?" It has to look at a combination of different status values, namely:

• manager_state should be idle
• items_in_queue should be 0.

This is an important question to ask the queueserver in an automated experiment/process to decide if a queue is really finished and we can move on the next thing.

However, there is an edge condition. Assume that there are 2 items in the queue. Once started the first item will be passed to the worker and the manager_state will go to something other than idle. Since there is no way to subscribe to changes to the queueserver, a client wanting to know if the queueserver is finished has to poll the queueserver. It does this asynchronously,

After the first item in the queue is finished the manager_state will momentarily go to idle while the RE worker is not doing anything and the final plan is loaded in from the queue. At this moment the items_in_queue will also be 0. A client looking at this status must conclude that the queueserver has finished since this combination of status (idle and items_in_queue ==0) is also the case when there is nothing left in the queue and everything is done. However, the final plan is loaded now into the RE and the manager_state will go for one last time into an non idle state.

This can be solved client side by getting the client to poll a number of times and "double check" that the manager_state is really idle also on the second check. However, this is a bit confusing.

Suggestion

1. Do not allow manager_state to be idle until there is nothing left in the queue and been executed
2. Allow some way to subscribe to changes and make status calls synchronous.

I am not sure of the implications of either of these, and wanted to discuss it here.