Support pre-removal of tasks to handle a future stall?

[hjoliver]

(Follow-on from #6383 [closed by #7248] which adds future final-incomplete tasks to n=0 for visibility.)

If I set a future task to (e.g.) failed or expired, such that it is in a final-incomplete state, the flow will eventually stall there.

One way of dealing with such a stall is to cylc remove the final-incomplete task - i.e., I say we need not continue the flow at that point, and I accept the downstream consequences of cutting it off there. In fact that is likely what I want to do if I've manually set the task to a final incomplete state (especially expired, which literally means there is no need to run the task anymore).

If I know in advance that the stall is coming, and I know that I will just remove the blocking task when it does come, I should be able to handle it in advance in the same way (without waiting till midnight Saturday or whatever).

Currently I can't do that, because removing the task ahead of the flow erases its history, so it will respawn when the flow gets there.

---

What to do about this?

We could have a cylc remove option to remove a task from n=0 without erasing its history, so that the scheduler knows it has been removed and should NOT respawn it in the same flow.

This is admittedly somewhat dangerous - pre-removing the final-incomplete task from n=0 may remove it from view in the UIs, which (at least until the current n-window extent covers it) means no visibility of the fact that the workflow may just shut down early when the flow reaches that task.

Regardless of that, IMO it is inarguable that I should be able to handle an impending stall before it happens, if I know it is impending and I know how I want to handle it, so we need some way to manage this sort of scenario.

A safe way to do it might be something like this: cylc remove --on-stall leaves the final-incomplete task in n=0 for now, but automatically removes it when/if it causes a stall? Possibly with some kind of attribute that marks the task as pre-removed?

[oliver-sanders]

So the use case is removing a subgraph of tasks ahead of the workflow.

Currently your approach is to set the task with an incomplete status, then wait for the flow to approach and all prerequsites to complete, then remove the task.

The approach we use at the MO for these scenarios is "skip mode":

• Tasks can be toggled on/off (i.e, you can flip-flop your decision to skip the tasks as many times as you like before they run).
• Skip mode tasks are indicated in the GUI with the skip symbol (i.e, you have visibility of the approaching skip-mode tasks as they enter the n-window).
• The skipped tasks will complete, so the workflow will not stall in future cycles as the result of any inter-cycle dependencies.
• Skip mode outputs can be customised if pre-prepared interventions are an option (e.g, branched graphs can be handled and recovery outputs can be configured for future cycles to trigger off of).
• Skip mode can be easily automated, so you may have a task in the workflow (or at the head of a chain) which decides whether a chain is needed or not and broadcasts to them as necessary.

[hjoliver]

Note I said "one way of dealing with such a stall" is to remove the blocking task - not the only way, and not necessarily "my approach" (although for this particular scenario, I do prefer it).

Removing a stall-causing task like this IS something that Cylc supports, and:

• it is an easier way to achieve the desired result
  • remove one task, vs work out what tasks are in the child-graph and set them all to skip mode
• it is more intuitive: if we don't need to run a task then why should we need to run its child-graph, even in skip mode?
  • (It's a simple leaf sub-graph that does not pragmatically require skip mode - otherwise I wouldn't expire the task)
• it is "safer" in one (not all!) respect(s): if I get the list of tasks wrong for skip mode, some will run live and fail

(Yes, the skip mode approach has pros too, but which is "best" depends on the situation and the user - we need both).

So, if that's the (perfectly legit) way I intend to handle the stall, and I know that the stall is impending, I should be able to do it without having to actually wait for the stall to occur, end of story.

---

Currently your approach is to set the task with an incomplete status, then wait for the flow to approach and all prerequsites to complete, then remove the task.

That makes it sound much more obtuse than it is 🤣. How about, I (or another task) determines that a future task no longer needs to be run, so I expire it - the intuitive thing to do because that's what expiry means as a concept. (And of course the point of this issue is I do not want to have to "wait for the flow to approach and all prerequisites to complete")

[hjoliver]

[As an aside, in the context of another current discussion, maybe we should consider "remove"-type (branch cutting) solutions as low-level interventions that should not appear in generic advice for the average user; but that doesn't mean we don't need them at all - they can allow experts to do things faster and to fix unusual problems - and besides, even for normal users the trade off between expertise, safety, and complexity is sometimes subjective and debatable. Here, is it REALLY easier for users to understand why they should have to arrange for an entire sub-graph to run in skip mode rather than just not run it at all, when it hangs off a task that they know does not need to run at all?].

[oliver-sanders]

it is an easier way to achieve the desired result
-- remove one task, vs work out what tasks are in the child-graph and set them all to skip mode

This is a circular argument:

• How do you know that you only need to remove one task?
  • To know this you require an understanding of the prerequisites of the sub-graph to be removed.
• How do you know the workflow won't stall downstream as a result of this operation?
  • To know this you require an understanding of the downstream prerequsites of the sub-graph.
• In order to know both the prerequisites of the subgraph and the tasks downstream of it, you must have knowledge of the whole subgraph.

And in the general case:

• What if there are multiple tasks connecting to the ones you wish to skip?
• What if there are multiple outputs spawning these tasks?

Not desperate to rake over this now, but to illustrate, this solution does require knowledge of the sub-graph! It's only "remove one task" because you already know which task to remove.

if we don't need to run a task then why should we need to run its child-graph, even in skip mode?
-- (It's a simple leaf sub-graph that does not pragmatically require skip mode - otherwise I wouldn't expire the task)

That's fair.

We choose to do this because completing these tasks produces fewer caveats than not running them (downstream stalls).

You can of course configure the output(s) skip-mode produces to taste, say to expire.

it is "safer" in one (not all!) respect(s): if I get the list of tasks wrong for skip mode, some will run live and fail

This argument equally plays against the remove approach:

• What if you remove the wrong task(s) and some downstreams run?
• What if a subsequent output re-spawns a removed task?

Debated to death on the other PR.

That makes it sound much more obtuse than it is 🤣

Using scheduler stall as the trigger to perform an intervention is somewhat obtuse!

What all of this, and also the --no-spawn discussion boils down to, is really the method we use for selecting tasks:

• [Future] remove-type solutions are flow-like operations:
  • We set in-motion a process which evolves the graph as it proceeds.
  • The affected tasks are implicitly determined by an advancing flow front.
• Set-type solutions are task-like operations:
  • We perform a single pool/DB operation.
  • The affected tasks are *explicitly specified.

So this is actually a pretty accurate model of what's happening when we cylc remove a task:

cylc trigger //remove-task --run-mode=None

(Assuming that the default --run-mode=live takes precedence over None in flow merging, the flow-front will cause on-path downstreams to spawn in as partially-satisfied.)

If I were to suggest a flow-like intervention for skip-mode, e.g:

cylc trigger //remove-task --run-mode=skip

We would probably consider that perverse!

(But functionally equivalent again assuming that the default --run-mode=live takes precedence over skip during flow-merging.)

we need both

We need a solution which suits all use cases.

But exactly what that solution should be is debatable. It doesn't have to be either cylc remove --on-stall or skip mode.

I wouldn't just jump straight to overloading the remove command. It's worth taking a step back, cylc remove is already complex enough as it is, and we're suggesting a solution which only suits certain situations, surely we can come up with something more intuitive?

[oliver-sanders]

The approach here is to set a final status on the task ("expired" would probably make the most sense), but this doesn't work because the task would be considered incomplete.

Obviously, this could be resolved by handling the :expire output in the graph:

pipeline-start => x => y = z
pipeline-start:expire?

At which point the intervention becomes: cylc set //remove-task -o expired, clean branch graphing, no workflow stall.

But I'm guessing you won't like that answer as the graph modification constitutes a pre-prepared intervention?

However, we could also solve this by modifying the task's completion condition. E.g, consider this broadcast command:

cylc broadcast <workflow> -p <cycle> -n remove-task
    -s 'completion = expired' \
    -s 'run mode = skip' \
    -s '[skip]outputs = expired'

Not a very snappy command, but highly effective, or at least it would be but I just discovered this bug - #7254.

• The task will be marked with the "skip" icon.
• It will expire as soon as the flow-front approaches.
• The broadcast can be recalled at any point until it spawns, and the broadcast can be recalled to allow the chain to be run if desired.

Another place where we solve this problem is in the "expire-trigger" feature, where or expire is automatically appended to the completion condition to tolerate expiry.

In theory, this sort of logic could be activated on-the-fly with something like cylc set -o expired --complete.

• The task would be expired.
• The --complete flag would then override the regular completion logic.
• The task would then be able to leave the pool naturally.

This also prevents the stall from happening in the first place.