warm start: handle interfaces which request pre-initial flow history

[oliver-sanders]

So called "warm starts" are an emergency fallback which allow us to restart a workflow from any point in the graph.

To achieve this:

1. Delete (or archive) the workflow DB.
2. Restart the workflow using the cylc play --start-cycle-point flag to specify the point in the graph that the workflow should continue from.
3. Cylc will assume that everything before this cycle has completed and continue running the workflow as normal.

This worked fine under Cylc 7, where all graph state was preserved in memory in the task pool. However, with Cylc 8, the state resides (partially) in the database. As a result, the pre-initial logic may say that a task has completed, whereas a database interface may say that it has not been spawned.

This inconsistency may result in undefined behaviours. So far we have encountered one example, where tasks triggered before the start cycle point will "flow-on" against user expectations (the user has told Cylc to assume that these tasks have completed, so would expect only the triggered tasks to run, not their downstreams).

Cylc should behave much the same in the warm start case as would in the cold start case (where the prior cycles had all completed).

As usage of the database to inform the task pool data window increases (#6143), there is potential for further conflicts.

As a temporary workaround for the issue of triggering pre-initial tasks, task spawning was restricted to cover only manually triggered tasks (#7148). However, this is an imperfect solution as, targetting only trigger, it breaks other expected behaviours.

Brute-force solution

On startup, fill in DB entries for all tasks between the initial and start cycle points (as if they had been completed in skip mode?).

Perfectly possible, perhaps the simplest and most reliable solution.

But also a tad ugly.

Wrap DB interfacing logic

E.g:

• If we try to spawn a task before the start cycle point AND there is no record of it in the database, then assume it has completed (pre-initial logic).
• Allow cylc remove to operate on no-record entries, demoting them to flow=None.
• Use the presence of a flow=None entry in the DB to indicate that a task should not be considered to have completed (allowing cylc remove and group-trigger to do their thing).
• Disallow cylc trigger --flow=none for pre-initial tasks (not needed since flow-on will be suppressed by default, avoids consistency issues).

[MetRonnie]

There is also currently the slight problem of the data store showing waiting tasks in the N-window on either side of a manually triggered pre-startcp task (and the N-window before the start cycle itself):

I had a poke at this and while a blanket ban on pre-startcp N-winddow tasks is easy enough, ideally we would only want to omit pre-startcp tasks if they have no history. Unfortunately, achieving this seems rather complicated to me, as the data store first fully generates all N-window parents and children, and then applies DB history if applicable. So we might have to prune the already-generated N-window parents/children if they have no history before the deltas are sent.

[hjoliver]

There is also currently the slight problem of the data store showing waiting tasks in the N-window on either side of a manually triggered pre-startcp task (and the N-window before the start cycle itself)

Currently the GUI shows the workflow as defined by the graph, and some unusual operations such as triggering tasks prior to the start cycle create constraints that aren't visually obvious in terms of downstream effects.

Perhaps ideally the GUI should reflect such constraints as well, but the question is how far can and should we go in attempting to do that? IMO that status quo is not unreasonable, and is at least easily explained.

(If we "removed" future waiting tasks because the triggered flow will terminate, what about the graph for new flows?)