Rebuild advanced visibility index from default datastore #7359
Description
Is your feature request related to a problem? Please describe.
Our organization has use cases where we need to create a sandbox environment based on data from another environment. E.g. we may wish to snapshot data in production and replicate it elsewhere to allow ops to test and debug failed workflows without affecting production itself. These source environments and the target environments need to be mutually isolated from each other once replication is completed.
The approach we are exploring is to create the target environment utilizing datastore backups from the source environment. We have been informed that due to the appropriate leveraging of transactions, the default datastore should be internally consistent. However, since advanced visibility is eventually consistent, it is likely that the default datastore and the advanced visibility datastore will be out of sync. We seek a standardized method of bringing the two stores in sync.
Proposed Solution
We request a feature to rebuild / repopulate advanced visibility indices. This should be able to accept some filters (datetime range, workflow whitelist / blacklist, etc.) and reproduce appropriate messages to kafka.
Additional context
Seems like we could leverage the existing Workflow Replay functionality to kickstart creating this feature