Core action to wait for trigger

[Rudd-O]

SUMMARY

Our workflows require us to wait for a specific trigger with specific parameters, sleeping indefinitely until that happens. There is no obvious way to do this with a Python action, and a Python action would eventually timeout anyway.

ISSUE TYPE

Pick one below and delete the rest:

• Feature Idea

STACKSTORM VERSION

3.0.1

STEPS TO REPRODUCE

Create workflow.
Observe there is no way to have the workflow pause until a condition (signaled by a trigger) is met.

[Rudd-O]

To be clear, I am not interested in triggering a new workflow -- the feature request is about suspending a workflow completely (whether through an action or some other way) until a trigger fires, and then I can run an action that can deal with the trigger next.

[Rudd-O]

Went thru the code and looked at the inquirer_runner.py file. That looks like we would need a new runner that simply pauses until a trigger event is received which matches the spec passed to the runner? Could it be implemented as an action? What I want is for the Orquesta workflow running said action to pause the branch executing the action, or heck, even continue running the action indefinitely with no timeout, until an event triggers it (or optionally a timeout has passed). Sorta like core.pause but monitoring the event queue and unsticking itself when the event happens.

The big problem with an action, specifically, is that when the st2actionrunner running the action dies (machine crashes), the Orquesta workflow just remains there, stuck forever, and in many cases can't even be canceled.

Please advise. I'd like to collaborate on a solution to this, but I don't know the internals of StackStorm well enough to figure this out.

[arm4b]

Sounds that https://docs.stackstorm.com/inquiries.html is the closest to what you want to do with Notifying Users of Inquiries using Rules, while event-queue like logic fits more into StackStorm sensors https://docs.stackstorm.com/sensors.html#sensors where you potentially can work with st2 direct calls/API.

Can you provide any simplified real-world example that would describe your use case and be valuable for the broader community?
Maybe one workflow that does it all could be divided into lower-level st2 primitives like several workflows/sensors/actions/inquiries instead of new runner type?

[LindsayHill]

The part I'm wondering about is how you connect the new trigger to the existing paused action. How do you map those together? What links them?

Speculating here, but what you could do is something like this:

1/ First workflow runs, gets to point where it pauses (using inquiries).
2/ New event received that should restart workflow.
3/ This event kicks off a workflow that somehow matches up that new event with the paused workflow (or takes a different action if there is no matching paused workflow).
4/ That new workflow finds the existing paused workflow, and responds to the inquiry, allowing it to pick up where it left off.

[Rudd-O]

Sort of. We have an external system that sends events to our platform (which we consume using sensors, which then send triggers). I can set up a rule to send those triggers to an action which gets executed on the spot.

The main problem I have is that my executions list gets polluted by all the actions that get triggered by the sensor's triggers. The secondary problem I have is that the actions triggered by the sensors cannot easily coalesce this way.

Speaking of how I worked around this problem -- seems like an awful workaround, by the way -- what I'm doing in this specific workflow is put a dead stop in the workflow in the form of an inquiry (exactly what you suggested). Then I have a separate action which gets executed by the trigger, which then (this is the very ugly part) internally queries all pending inquiries, guesses (based on the trigger payload) if they need to be answered in the affirmative, and then answers in the affirmative if they match the guess. This, of course, requires an awful lot of code (most of which is using the st2 pack library code, because there's no inbuilt functionality in StackStorm to call other packs' actions from your custom pack's action).

So, to sum it up, here's the workaround.

• Workflow you execute does A -> B -> C where B is an inquiry. B gets to state pending after A is done.
• Sensor S dispatches trigger payload T.
• Action R reacts to payload T, queries existing pending inquiries, finds B, looks into its schema, retrieves data from the schema, compares it with payload T -- if it matches, then R responds to the inquiry B.
• This unlocks B with a certain data payload answer to B's question.
• B continues to C.
• C can then consume data from what B published, and proceed.

This seems excruciatingly painful to me, as well as super hard to compose. If we just had an action st2.wait_for_trigger that would just wait for the trigger (and was pausable), and then I could write an action that processes the output of the trigger, then I'm game. That'd be much better. It would, of course, be far better than cluttering the UI with silly questions that would get answered by the computer, or silly workaround actions that were answering as the computer.

[LindsayHill]

(most of which is using the st2 pack library code, because there's no inbuilt functionality in StackStorm to call other packs' actions from your custom pack's action).

Could you explain a bit more about what you mean here? Normal design pattern is to have a workflow in your custom pack, and that workflow calls actions from other packs. Or are you talking about writing a custom Python action and within that action calling other actions?

[Rudd-O]

Well, we have a long-running workflow (1 month or more) which iterates over a bunch of hosts that need some work done. This work cannot start until an external system notifies us (via a sensor) that so-and-so hosts are ready to go. Which is fine, we have the sensor and we can bind it to some action without much fuss using a rule.

In our workflow, for now, we have an inquiry that basically freezes the workflow until someone says "yes". We've also built a separate "robot finger" action which we have bound to our sensor via a rule, to hit "yes" on the inquiry, and then the workflow can proceed, then return to the inquiry until there are no more hosts in the pending list. The robot finger scans all workflows that are paused with a pending inquiry, and responds the inquiry -- that's basically how we get the workflow to continue when it's appropriate.

But we're talking thousands of invocations, so the list of actions gets hugely polluted with this action getting called by the sensor / rule.

It would be far better if we had an action we can integrate into our workflow, which pauses the workflow, and then the rule can unpause the workflow. Thus we don't invoke that "robot finger
action at all.

[LindsayHill]

How does the rule figure out which workflow should be unpaused? How do you connect the two?

The robot finger scans all workflows that are paused with a pending inquiry, and responds the inquiry -- that's basically how we get the workflow to continue when it's appropriate.

How do you plan to get around this? How would you connect the sensor event to the paused action?

It would be far better if we had an action we can integrate into our workflow, which pauses the workflow, and then the rule can unpause the workflow. Thus we don't invoke that "robot finger
action at all.

Are you aware that you use core.pause to pause an action, and st2 execution resume <ID> to resume a paused action? You don't have to use Inquiries.

You might be over-thinking this "robot finger" thing - inquiries are just another action, just another API. They don't have to be human-driven.

[Rudd-O]

In my current -- very suboptimal -- setup, the rule answers the core.ask in the workflow by invoking a separate task. Said task already knows when it has to answer, and when it shouldn't (thereby not unpausing workflows that do not need unpausing). This works fine.

Further problems:

I tried running in my workflow two parallel tasks (1) a core.ask (2) another task that, if the condition is true, answers the core.ask via the API. Well, there's a race condition: the core.ask action pauses the entire workflow, not just its branch, so (2) never gets to run...

I think that is a legit bug. Core.ask should only block dependent tasks, not the entire thing which may have parallel branches running and should not stop.

Aaaargh. Any ideas?

[Rudd-O]

To illustrate:

• caller workflow begins
  • workflow begins
  • in parallel:
    • wait for events from database pertaining object X (using core.ask)
    • query database to see if object X has already had its event
      • if query positive, answer sibling core.ask
      • else, do nothing more
  • return to caller workflow
• do stuff

So, if the query returns negative, core.ask remains waiting until the trigger calls the action that tells core.ask "you may continue". Then the trigger happens sometime in the future. Then core.ask gets answered, then workflow returns to parent workflow, then continues to "do stuff".

In other words: sensor trigger -> core.ask respond works fine.

The problem is when the query returns positive, and core.ask respond from the same workflow happens. Sometimes this works. Sometimes absurd things happen -- the pause workflow code in core.ask pauses the workflow while the respond task is running, which results in a workflow that has a successfully answered core.ask, BUT the parent is paused! This is illogical.

There is a race condition whereby the core.ask code responsible for pausing the workflow runs while the API call to respond the core.ask takes place. As a result, the workflow is paused even when then core.ask is successfully answered. WTF.

[Rudd-O]

FYI TTL parameter in core.ask does not do anything. I set it to 1 minute and 1 minute has passed... nothing.

[LindsayHill]

The TTL parameter does work, I have tested it recently.

You need to enable Garbage Collection for Inquiries, and if you require very short expiries, you may need to change your garbage collection interval