Orchestration ID reuse policies

[cgillum]

This idea comes from dapr/dapr#7101, as it relates to reusing existing workflow IDs.

When scheduling the creation of a new orchestration, the following options should be available:

• TerminateIfExists - Terminates any existing orchestrations with the same instance ID and then schedules a new instance as one atomic action, similar to on-demand ContinueAsNew.
• SkipIfExists - If there is an existing orchestration already scheduled, then the scheduler does nothing.
• ThrowIfExists - If there is an existing orchestration then the scheduler throws an exception (this is the current behavior).

As an optional extension of this, we can consider exposing an "Execution ID" or "Generation ID" property to the orchestration which can help distinguish different instantiations of the given instance ID.

[clintsinger]

My vote for having the extra ID to be able to identify different runs of the workflow in logs.

[olitomlinson]

@cgillum

Would it be reasonable to add a hint of : SKIP_IF_RUNNING_OR_COMPLETE_OR_PAUSED

An example, scenarios where workflows are being started via queue with at-least-once guarantees (which is quite common) it would be useful to have this hint because :

• It would prevent a workflow being prematurely terminated & rescheduled if it was currently in a running state.
  • Restarting a running workflow may be wasteful and incur unwanted / unanticipated side-effects when repeated from the start.
• It would prevent a workflow being rescheduled if it was in a complete state
  • Supporting idempotent design - Some business processes should never be repeated once they've ran to completion
• It would prevent a workflow being rescheduled if it was in a paused state
  • Pausing a workflow would be a deliberate human action. Therefor automatically restarting it may cause undesirable side-effects.
• However, it would allow a workflow in a failed state to be rescheduled, which could be desirable.

Unless I'm mistaken, It's my understanding that none of the 3 new reuse policies would accommodate the above scenario?

[clintsinger]

The running state is exactly the one I want to interrupt for the scenario that I originally proposed. I agree that once it is complete then rescheduling is just a matter of starting a new instance.

The original definition of TerminateIfExists didn't care if it was just scheduled or already running. The workflow would be terminated. SkipIfExists wouldn't care either way either.

[olitomlinson]

Yes this is why I'm proposing an additional policy, so that it doesn't interfere with the needs of your use-case (which I think is TERMINATE_IF_EXISTS)

[clintsinger]

So you are suggesting that one combines, for example, TERMINATE_IF_EXISTS and SKIP_IF_RUNNING_OR_COMPLETE?

As a way to differentiate what to do if it is just scheduled vs already running?

[clintsinger]

I am assuming this doesn't have much meaning for SKIP_IF_EXISTS and SKIP_IF_RUNNING_OR_COMPLETE since it would already be skipped from the first policy.

[olitomlinson]

This is how I interpret the current proposal :

Start = start/restart the orchestration
No-op = leave the orchestration in its current state (do not start/restart)

	Doesn't Exist	Running	Completed	Failed	Paused
TERMINATE_IF_EXISTS	Start	Start	Start	Start	Start
THROW_IF_EXISTS	Start	No-op	No-op	No-op	No-op
SKIP_IF_EXISTS	Start	No-op	No-op	No-op	No-op

Assuming my interpretation above is correct, I would like to see the following policy added SKIP_IF_RUNNING_OR_COMPLETE_OR_PAUSED

	Doesn't Exist	Running	Completed	Failed	Paused
SKIP_IF_RUNNING_OR_COMPLETE_OR_PAUSED	Start	No-op	No-op	Start	No-op

[kaibocai]

Thanks @olitomlinson for the explanation on SKIP_IF_RUNNING_OR_COMPLETE_OR_PAUSED I think it makes sense to support idempotent design. @cgillum what do you think about this OPTION?

[cgillum]

Catching up on this discussion. If I understand the proposal from @olitomlinson, the problem you're trying to solve is that you want to only reuse the orchestration ID if an existing instance is in a failed state. I think that makes sense. Would it just be "failed", or could it also include "Terminated" (and perhaps some future "Canceled" state)?

[clintsinger]

Wouldn't it be easier just to say RESTART_IF_FAILED (or RESCHEDULE_IF_FAILED)?

It appears to only be a rule for when the workflow is in a failed state.

[olitomlinson]

@cgillum I agree, it should be SKIP_IF_RUNNING_OR_COMPLETE_OR_PAUSED_OR_TERMINATED - I would bucket paused and terminate together as they are both deliberate human actions, which shouldn't be automatically restarted.

Although I think going down this path is going to lead to a trying to support a huge set of highly specific, Policies which isn't sustainable.

Suggestion

Divorce the ACTION (Terminate, Skip, Throw) from the applicable STATE/S, such that a user can dial in the exact requirement.

TERMINATE_IF_EXISTS would be expressed as...

WorkflowOptions { 
    ReusePolicy = new ReusePolicy(action: Action.Terminate, states: new[]{ State.All })
}

SKIP_IF_RUNNING_OR_COMPLETE_OR_PAUSED_OR_TERMINATED would be expressed as...

WorkflowOptions { 
    ReusePolicy  = new ReusePolicy(action: Action.Skip, states: new[]{ State.Running, State.Complete, State.Paused, State.Terminated})
}

[clintsinger]

I'd recommend using enum flags for the states field.