Signal loss during Redis ContinueAsNew transition

[jsaevecke]

Problem

When a workflow uses ContinueAsNew, the old execution is replaced by a new execution with the same InstanceID but a new ExecutionID.

Signals are sent to the currently active execution.

With the Redis backend, there is a race where a signal can still be written to the old execution after the new execution has already become active.

Example

Assume workflow instance A currently runs as execution E1.

1. SignalWorkflow("A") looks up the active execution and finds E1.
2. Before the signal is written, E1 completes with ContinueAsNew.
3. A new execution E2 is created and becomes the active execution for A.
4. The signal operation continues and writes the signal to E1.

At this point the signal is stored on the old execution, while E2 is now the active execution.

The signal is therefore not processed by the continued workflow.

Expected Behavior

A signal that races with ContinueAsNew should not be lost.

It should either:

• reach E1 before the transition and be carried over to E2, or
• be delivered directly to E2.

Backend

Redis

Possible Fix

I have a possible fix that makes active-execution lookup and signal delivery atomic and also handles pending events during the ContinueAsNew transition.

I would be happy to submit a PR if this approach is acceptable.