backup: add two-phase retry mechanism

This commit teaches backup to use a two-phase retry raather than the
attempt-based retry we use today. Under this new retry mechanism, backup
will use an attempt based retry until all nodes have reported progress.
After all nodes have done so, we switch to a short duration based policy
that is reset each time progress is made.

Resolves: cockroachdb#159388

Release note: None

Author: kev-cao

pkg/backup/BUILD.bazel

@@ -135,6 +135,7 @@ go_library(
         "//pkg/util/admission",
         "//pkg/util/admission/admissionpb",
         "//pkg/util/besteffort",
+        "//pkg/util/buildutil",
         "//pkg/util/bulk",
         "//pkg/util/ctxgroup",
         "//pkg/util/envutil",