br: Return the timestamp of backup to cooperate with TiKV-CDC

[pingyu]

Feature Request

Is your feature request related to a problem? Please describe:

TiKV-BR is designed to initialize data before the incremental data replication of TiKV-CDC. Refer to RawKV Cross Cluster Replication (tikv/rfcs#86).

To cooperate with TiKV-CDC, the procedure is expected to be:

1. TiKV-BR backup from upstream TiKV
   1.1 Get a latest TSO (supposed to be T0).
   1.2 Set BR service safepoint to T0 and TTL as 72 hours (configurable).
   1.3 Invoke backup procedure to TiKV, flush causal_ts at the beginning to make sure that timestamp of following writes must be greater than T0.
   1.4 Return the T0 after backup finished.
2. TiKV-BR restore to downstream TiKV (It would be nice the store T0 in backup meta to remind the users agin after restore).
3. TiKV-CDC create changefeed between upstream & downstream with --start-ts T0.

Update 2022.6.29

Set T0 as TSO - safe-interval. Where safe-interval is a configurable duration defaults to 1 minute.

safe-interval is a work around to avoid missing some RawKV writes just in Raft procedure during backup, and assume that all Raw writes should be finished in 1 minute.
The cons of safe-interval is that some entries would be backup or replicated redundantly.
On a system with heavy load, the safe-interval should be configured longer.
The safe-interval would be deprecated after the read-ts of RawKV is implemented. Refer to tikv/rfcs#96.

Describe the feature you'd like:

1.1 ~ 1.4 in above description.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

[pingyu]

cc @sunxiaoguang @haojinming @zeminzhou @ystaticy , PTAL~

[haojinming]

There is gap between T0 and the time(suppose to be T1) to flush_time_stamp at the beginning of backup. If some data are written between T0 and T1, the timestamp in key maybe smaller than T0, then the data may not be backuped by BR and captured by CDC.

[pingyu]

Emm you are right, I miss this.

Actually I think this is the issue that how we implement backup in a running TiKV. (Incremental backup also need this).
I would like to limit this feature to a static backup (or backup for a static TiKV) now.

p.s. This scenario is very similar with CDC: How we know there is no data in the fly, or what's the minimal timestamp of the flying data. The resolved-ts may help us to solve this problem. But we need to extend it to not-subscribed regions.