@@ -132,7 +132,7 @@ T = 0: adding event with event ID == 1 & version == 1
132132View in both Cluster A & B
133133
134134```
135- | -------- | ------------- | --------------- | ------- |
135+ | -------- | --------------- | --------------- | ------- |
136136| Events | Version History |
137137| -------- | --------------- | --------------- | ------- |
138138| Event ID | Event Version | Event ID | Version |
@@ -146,7 +146,7 @@ T = 1: adding event with event ID == 2 & version == 1
146146View in both Cluster A & B
147147
148148```
149- | -------- | ------------- | --------------- | ------- |
149+ | -------- | --------------- | --------------- | ------- |
150150| Events | Version History |
151151| -------- | --------------- | --------------- | ------- |
152152| Event ID | Event Version | Event ID | Version |
@@ -161,7 +161,7 @@ T = 2: adding event with event ID == 3 & version == 1
161161View in both Cluster A & B
162162
163163```
164- | -------- | ------------- | --------------- | ------- |
164+ | -------- | --------------- | --------------- | ------- |
165165| Events | Version History |
166166| -------- | --------------- | --------------- | ------- |
167167| Event ID | Event Version | Event ID | Version |
@@ -178,7 +178,7 @@ adding event with event ID == 4 & version == 2
178178View in both Cluster A & B
179179
180180```
181- | -------- | ------------- | --------------- | ------- |
181+ | -------- | --------------- | --------------- | ------- |
182182| Events | Version History |
183183| -------- | --------------- | --------------- | ------- |
184184| Event ID | Event Version | Event ID | Version |
@@ -195,7 +195,7 @@ T = 4: adding event with event ID == 5 & version == 2
195195View in both Cluster A & B
196196
197197```
198- | -------- | ------------- | --------------- | ------- |
198+ | -------- | --------------- | --------------- | ------- |
199199| Events | Version History |
200200| -------- | --------------- | --------------- | ------- |
201201| Event ID | Event Version | Event ID | Version |
@@ -229,7 +229,7 @@ T = 0:
229229View in both Cluster B & C
230230
231231```
232- | -------- | ------------- | --------------- | ------- |
232+ | -------- | --------------- | --------------- | ------- |
233233| Events | Version History |
234234| -------- | --------------- | --------------- | ------- |
235235| Event ID | Event Version | Event ID | Version |
@@ -243,7 +243,7 @@ View in both Cluster B & C
243243T = 1: adding event with event ID == 4 & version == 2 in Cluster B
244244
245245```
246- | -------- | ------------- | --------------- | ------- |
246+ | -------- | --------------- | --------------- | ------- |
247247| Events | Version History |
248248| -------- | --------------- | --------------- | ------- |
249249| Event ID | Event Version | Event ID | Version |
@@ -258,7 +258,7 @@ T = 1: adding event with event ID == 4 & version == 2 in Cluster B
258258T = 1: namespace failover to Cluster C, adding event with event ID == 4 & version == 3 in Cluster C
259259
260260```
261- | -------- | ------------- | --------------- | ------- |
261+ | -------- | --------------- | --------------- | ------- |
262262| Events | Version History |
263263| -------- | --------------- | --------------- | ------- |
264264| Event ID | Event Version | Event ID | Version |
@@ -275,40 +275,38 @@ T = 2: replication task from Cluster C arrives in Cluster B
275275Note: below are a tree structures
276276
277277```
278- | -------- | ------------- |
279- | Events |
280- | ------------- | ------------- |
281- | Event ID | Event Version |
282- | -------- | ------------- |
283- | 1 | 1 |
284- | 2 | 1 |
285- | 3 | 2 |
286- | -------- | ------------- |
287- | |
288- | ------------- | ------------ |
289- | |
290- | -------- | ------------- | | -------- | ------------- |
291- | Event ID | Event Version | | Event ID | Event Version |
292- | -------- | ------------- | | -------- | ------------- |
293- | 4 | 2 | | 4 | 3 |
294- | -------- | ------------- | | -------- | ------------- |
295-
296- | --------------- | ------- |
297- | Version History |
298- | --------------- | ------------------- |
299- | Event ID | Version |
300- | --------------- | ------- |
301- | 2 | 1 |
302- | 3 | 2 |
303- | --------------- | ------- |
304- | |
305- | ------- | ------------------- |
306- | |
307- | --------------- | ------- | | --------------- | ------- |
308- | Event ID | Version | | Event ID | Version |
309- | --------------- | ------- | | --------------- | ------- |
310- | 4 | 2 | | 4 | 3 |
311- | --------------- | ------- | | --------------- | ------- |
278+ | ------------- | ------------- |
279+ | Events |
280+ | ------------- | ------------- |
281+ | Event ID | Event Version |
282+ | ------------- | ------------- |
283+ | 1 | 1 |
284+ | 2 | 1 |
285+ | 3 | 2 |
286+ | -------- | ------------- |
287+ | |
288+ | |
289+ | -------- | ------------- | | -------- | ------------- |
290+ | Event ID | Event Version | | Event ID | Event Version |
291+ | -------- | ------------- | | -------- | ------------- |
292+ | 4 | 2 | | 4 | 3 |
293+ | -------- | ------------- | | -------- | ------------- |
294+
295+ | --------------- | ------------------- |
296+ | Version History |
297+ | --------------- | ------------------- |
298+ | Event ID | Version |
299+ | --------------- | ------- |
300+ | 2 | 1 |
301+ | 3 | 2 |
302+ | --------------- | ------------------- |
303+ | |
304+ | |
305+ | --------------- | ------- | | --------------- | ------- |
306+ | Event ID | Version | | Event ID | Version |
307+ | --------------- | ------- | | --------------- | ------- |
308+ | 4 | 2 | | 4 | 3 |
309+ | --------------- | ------- | | --------------- | ------- |
312310```
313311
314312T = 2: replication task from Cluster B arrives in Cluster C, same as above
@@ -345,27 +343,22 @@ Multi-cluster Replication aims to keep the Workflow Execution History as up-to-d
345343Due to the nature of Multi-cluster Replication (for example, Workflow Execution History events are replicated asynchronously) different Runs (same Namespace and Workflow Id) can arrive at the target Cluster at different times, sometimes out of order, as shown below:
346344
347345```
348- | ------------- | | ------------- | | ------------- |
349- | Cluster A | | Network Layer | | Cluster B |
350- | --------- || ------------- | | ------------- |
351- | | |
352- | Run 1 Replication Events | |
353- | -----------------------> | |
354- | | |
355- | Run 2 Replication Events | |
356- | -----------------------> | |
357- | | |
358- | | |
359- | | |
360- | | Run 2 Replication Events |
361- | | -----------------------> |
362- | | |
363- | | Run 1 Replication Events |
364- | | -----------------------> |
365- | | |
366- | --- || ------------- | | ------------- |
367- | Cluster A | | Network Layer | | Cluster B |
368- | --------- || ------------- | | ------------- |
346+ +-----------+ +----------------+ +-----------+
347+ | Cluster A | | Network Layer | | Cluster B |
348+ +-----------+ +----------------+ +-----------+
349+ | | |
350+ | Run 1 Replication | |
351+ |---------------------> | |
352+ | | |
353+ | Run 2 Replication | |
354+ |---------------------> | |
355+ | | |
356+ | | Run 2 Replication |
357+ | |---------------------> |
358+ | | |
359+ | | Run 1 Replication |
360+ | |---------------------> |
361+ | | |
369362```
370363
371364Because Run 2 appears in Cluster B first, Run 1 cannot be replicated as "runnable" due to the rule ` at most one Run open ` (see above), thus the "zombie" Workflow Execution state is introduced.
@@ -407,23 +400,23 @@ T = 0: task A is generated according to Event Id: 4, version: 2
407400T = 1: conflict resolution happens, Workflow Execution's mutable state is rebuilt and history Event Id: 4, version: 3 is written down to persistence
408401
409402```
410- | -------- | ------------- |
411- | Events |
412- | ------------- | ------------------------------- ------------- |
413- | Event ID | Event Version |
414- | -------- | ------------- |
415- | 1 | 1 |
416- | 2 | 1 |
417- | 3 | 2 |
418- | -------- | ------------- |
419- | |
420- | ------------- | ------------------------------- ------------- |
421- | |
422- | -------- | ------------- | | -------- | ------------- |
423- | Event ID | Event Version | | Event ID | Event Version |
424- | -------- | ------------- | | -------- | ------------- |
425- | 4 | 2 | <-- task A belongs to this event | 4 | 3 | <-- current branch / mutable state |
426- | -------- | ------------- | | -------- | ------------- |
403+ | --------- | ------------- |
404+ | Events |
405+ | --------- | ------------- |
406+ | Event ID | Event Version |
407+ | -------- | ------------- |
408+ | 1 | 1 |
409+ | 2 | 1 |
410+ | 3 | 2 |
411+ | -------- | ------------- |
412+ | |
413+ | --------- | ------------- |
414+ | |
415+ | -------- | ------------- | | -------- | ------------- |
416+ | Event ID | Event Version | | Event ID | Event Version |
417+ | -------- | ------------- | | -------- | ------------- |
418+ | 4 | 2 | <-- task A belongs to this event | 4 | 3 | <-- current branch / mutable state
419+ | -------- | ------------- | | -------- | ------------- |
427420```
428421
429422T = 2: task A is loaded.
0 commit comments