Race condition in workflow tester causes TM_WallClock mode for instant mocked activities

[nikitin-n]

Hi,

I get flaky unit-tests using workflowTester. They appears during slow cpu-restricted gitlab-runner and never (?) locally. I guess the reason is a race condition in https://github.com/cschleiden/go-workflows/blob/f14c4f3fe57162f5e9b1c1743cfa44b74085d42b/tester/tester.go

The newTimerMode() method (tester.go:547-554) may incorrectly return TM_WallClock for instantly-completed mocked activities due to the race

Activity goroutine (scheduleActivity, tester.go:651):

// scheduleActivity (L651):
atomic.AddInt32(&wt.runningActivities, 1)    // L654
go func() {
    defer atomic.AddInt32(&wt.runningActivities, -1) // L657: runs AFTER return
    // ... mocked activity executes instantly ...
    wt.callbacks <- callback                  // L726: non-blocking, goroutine is still alive
}()

...
...

// newTimerMode (L547):
runningActivities := atomic.LoadInt32(&wt.runningActivities) // L548: still > 0!
if runningActivities > 0 {
    return TM_WallClock                      // L550: wrong if goroutine defer (L657) hasn't run yet
}

Result: timer is scheduled via wallClock.AfterFunc(remainingTime) (L532) for real wall-clock time instead of firing instantly. After TestTimeout (10s) get panic: No new events generated during workflow execution and no pending timers, workflow blocked?

[nikitin-n]

For now I have no idea how to fix it properly.
Maybe we should move defer atomic.AddInt32(&wt.runningActivities, -1) inside callback:

wt.callbacks <- func() *history.WorkflowEvent {
        atomic.AddInt32(&wt.runningActivities, -1)
         ...
}