time-skipping integration with cross-cluster replication

[feiyang3cat]

What changed?

1. time skipping regenerates timer tasks in an idempotent way
2. calling time-skipping regeneration in PartialRefresh
3. add logic of passive timer queues for time skipping timer tasks

Why?

the goal is to make sure time skipping works correctly under all replication patterns (state-based, event-based)

How did you test it?

• built
• run locally and tested manually
• covered by existing tests
• added new unit test(s)
• added new functional test(s)

[feiyang3cat]

RegenerateTimerTasksForTimeSkipping is made idempotent for replication purpose

[feiyang3cat]

@yux0 @robholland
right now, when closeTransaction, the active cluster calls regenerateTimerTasks.
And should regenerateTimerTasks be called for passive cluster as well so that event-state based replication can work correctly?

[feiyang3cat]

this is used for state-based replication

[feiyang3cat]

in the active cluster, this TimeSkippingTimerTask will check time-skipping should be disabled (the fields are in mutable state); it seems to me in the passive cluster, it shall be noop OR something similar to executeUserTimerTimeoutTask with processTimer?

[feiyang3cat]

Question for @yux0 @robholland :

1. for state-based replication: partial refresh is called so the timer tasks are generated
2. for event-based replication: how could the timer tasks be generated
   (currently, the ApplyXXXEvents doesn't generate timer tasks in the active cluster and the active cluster calls RegenerateTimerTasksForTimeSkipping at the time the time of closeTransaction -> prepareTasks

and should we call RegenerateTimerTasksForTimeSkipping both for Active and Passive policy?

func (ms *MutableStateImpl) closeTransactionRegenerateTimerTasksForTimeSkipping(
	transactionPolicy historyi.TransactionPolicy,
) error {
	switch transactionPolicy {
	case historyi.TransactionPolicyActive:
		if !ms.IsWorkflowExecutionRunning() {
			return nil
		}
		return ms.taskGenerator.RegenerateTimerTasksForTimeSkipping()
	case historyi.TransactionPolicyPassive:
		return nil
	default:
		return serviceerror.NewInternalf("unknown transaction policy: %v", transactionPolicy)
	}
}

[yycptt]

I am still unclear on why we need to support event-based replication for a new feature.

State-based replication is fully enabled in cloud. Enabled by default in oss, so when this feature is available, oss users will be using state-based replication as well. S2C migration for any new release (>= 1.32) have to use state-based replication as well given the potential usage of chasm executions (which will be enabled by default).