xtriggers: implement as a run mode

[oliver-sanders]

Implement xtriggers as a task "run mode" rather than a scheduler service.

(write up of #3497 (comment))

Xtriggers Vs Polling Tasks

At present, xtriggers are essentially syntax sugar for polling tasks.

E.g this:

[scheduling]
  [[xtriggers]]
    poller1 = poller(file="file1", cycle="%(cycle)s", sequential=False)
    poller2 = poller(file="file2", cycle="%(cycle)s", sequential=False)
  [[graph]]
    P1D = @poller1 => x
    P1M = @poller2 => y

Is equivalent to this:

[scheduling]
  [[graph]]
    P1D = poller1 => x
    P1M = poller2[-P1M] => poller2 => y

At present, poller tasks can do everything that xtriggers can, but xtriggers (as a more simplified abstraction) cannot do everything that polling tasks, e.g, some graphing limitations of xtriggers:

# conditional expressions
@a | @b => c

# expire-triggers
@x => x & !@y & !y
@y => y & !@x & !x

# multiple outputs
@poller:file1 => x
@poller:file2 => y
@poller:file3 => z

Problems With The Status Quo

Because xtriggers are a parallel implementation to tasks, we have to implement much of the infrastructure they require in parallel too.

E.g:

• Bespoke interfaces for viewing and setting xtrigger prerequistes (duplicates task prerequisites).
• Bespoke sequential spawning mechanism (alternative non SoD spawning mechanism).
• Bespoke interface for xtrigger configuration (duplicates [runtime] functionality).

This causes maintenance burden and makes the situation more complex for users who need to learn the nuances of these two parallel systems. It would be simpler if we could reduce this down to a single implementation.

But beyond that, it also means that xtriggers are not able take advantage of task features which might be desirable.

• xtriggers: tree view and beyond cylc-ui#331
• xtriggers (graph view) cylc-ui#131
• Show external trigger values in the GUI. cylc-ui#49
• xtrigger timeout #6319
• data-store: incorrect information for n>0 xtriggers #6298 (xtriggers: differentiate "not satisfied" from "not yet evaluated" #6560)
• disable or replace xtriggers in sim modes #6201
• xtriggers: measure average runtime #5053
• tasks waiting on xtriggers etc. are n=1 #4012
• xtrigger housekeeping tweak? #7027
• Potential minor xtrigger improvements #6750
• Force un/complete xtriggers #6725
• custom xtriggers: (suicide-like?) "dead" function clean up #3232
• parameters: xtriggers #3454

We could attempt to develop a bespoke implementation to satisfy each of these missing interfaces, or, we could implement xtriggers as tasks and get all of this for free.

Xtriggers As Tasks

Basically, keep xtriggers functionally the same, but change the way they are defined / implemented so we can remove the bespoke interfaces straightening out the internals of Cylc.

We now have the "run mode" abstraction in Cylc which means that we can develop new task submission methods. E.g, it would be possible to implement async xtriggers as a "run mode".

Configure the task to use this run_mode and it automatically becomes an xtrigger.

I.e, this:

[scheduling]
  [[xtriggers]]
    poll_foo = poller(cycle="%(cycle)s", file="foo")
  [[graph]]
    P1D = @poll_foo => bar

Could be turned into this:

[scheduling]
  [[graph]]
    P1D = @poll_foo => bar

[runtime]
  [[@poll_foo]]
    run_mode = xtrigger
    script = poller
    [[[environment]]]
        FILE = foo

Unlocked Workflow Design Patterns

Chained xtriggers:

@foo => @bar => baz

Graph branching:

@foo | @bar => baz

Control over xtrigger spawning:

@wall_clock => @foo => bar

$ext-trigger => @foo => bar

Xtrigger expiry:

[scheduling]
  [[special tasks]]
    expire-triggers = @foo
  [[graph]]
    P1D = @foo => bar

Xtrigger time limits:

[runtime]
  [[@foo]]
    run mode = xtrigger
    execution time limit = PT1H

Configurable xtriggers:

$ cylc broadcast <workflow> -n POLLERS -s '[environment]PLATFORM = hpc-live'

Multi-file pollers:

@poller:file1 => file1
@poller:file2 => file2
@poller:file3? => file3

[hjoliver]

At present, xtriggers are essentially syntax sugar for polling tasks.
...
At present, poller tasks can do everything that xtriggers can, but xtriggers (as a more simplified abstraction) cannot do everything that polling tasks, e.g, some graphing limitations of xtriggers:

Syntactically yes, largely, but xtriggers do have a couple of functional advantages over polling tasks:

• Live xtriggers are shared to the full extent possible: if many tasks depend on the same one (as measured by function signature) only one call will be made, and additionally the same call will not be made again until the previous one returns.
• Workflow, cycle, task name, and task ID specificity: https://cylc.github.io/cylc-doc/stable/html/user-guide/writing-workflows/external-triggers.html#task-and-cycle-specificity

(How much these matter depends, of course).

[hjoliver]

We could attempt to develop a bespoke implementation to satisfy each of these missing interfaces, or, we could implement xtriggers as tasks and get all of this for free.

I certainly agree this is a HUGE argument in favour. And it probably outweighs any advantages of current xtriggers.

[oliver-sanders]

Syntactically yes, largely, but xtriggers do have a couple of functional advantages over polling tasks:

I'm not quite sure I follow on these?

But I am aware of one functional advantage of xtriggers, which is that they provide a very easy method of metadata communication - return True, data - which gets turned into broadcasts, this is simpler than configuring broadcasts manually.

Live xtriggers are shared to the full extent possible: if many tasks depend on the same one (as measured by function signature) only one call will be made, and additionally the same call will not be made again until the previous one returns.

I'm not sure how xtrigger "sharing" is different to task "triggering"?

Xtriggers and polling tasks are both "shared" via graph triggering, e.g, @x_poller and t_poller in this example will be called in the same way and behave the same in general:

[scheduling]
  [[xtriggers]]
    x_poller = poll(...):PT30S
  [[graph]
    P1D = """
      @x_poller => a
      @x_poller => b

      t_poller => c
      t_poller => d
    """
[runtime]
  [[t_poller]]
    script = """
      while true; done
        python poll.py
        sleep 30
     done
    """

Workflow, cycle, task name, and task ID specificity: https://cylc.github.io/cylc-doc/stable/html/user-guide/writing-workflows/external-triggers.html#task-and-cycle-specificity

I'm not sure xtrigger specificity is different to task scheduling?

Workflow specificity:

xtrigger = fcn(...)
# or:
R1 = poller
P1D = poller[^] => foo

Cycle specifity:

xtrigger = fcn(..., cycle=%(point)s)
# or:
P1D = poller => foo

Task specifity:

P1D = """
xtrigger_a = fcn(..., cycle=%(point)s, task=a)
xtrigger_b = fcn(..., cycle=%(point)s, task=b)
# or:
  a_poller => a
  b_poller => b
"""