K8SPS-402: allow parallel backups

[pooknull]

https://perconadev.atlassian.net/browse/K8SPS-402

DESCRIPTION

Problem:
The ticket mentions that we should implement the same functionality as in the https://perconadev.atlassian.net/browse/K8SPXC-1366.

Solution:

1. New allowParallel field

This PR adds spec.backup.allowParallel to the cluster

allowParallel controls whether multiple backups for the same cluster may run at the same time

• When omitted or set to false, the operator uses a backup lease allowing only one backup to run at a time
• When set to true, the operator does not use the lease, allowing multiple backup Jobs to run concurrently

2. New startingDeadlineSeconds field

This PR adds spec.backup.startingDeadlineSeconds to the cluster and spec.startingDeadlineSeconds to backup CRs

startingDeadlineSeconds defines how long a backup may wait before its job is created.

The deadline is measured from the backup creation time while the backup remains in the New or Starting state without a job.

• If the cluster becomes ready and the job is created before the deadline, the backup proceeds normally.
• If the deadline expires before the job is created, the operator marks the backup as Error and releases its backup lease.

With these changes operator will keep the backup pending if cluster is unready until it becomes ready or startingDeadlineSeconds expires, instead of setting it to Error state

3. New suspendedDeadlineSeconds field

This PR adds spec.backup.suspendedDeadlineSeconds to the cluster and spec.suspendedDeadlineSeconds to backup CRs

suspendedDeadlineSeconds defines how long a suspended backup job may wait for the cluster to become ready again.

When the backup is active and cluster becomes non-ready, the operator suspends the backup job, changes the backup state to Suspended, and starts measuring the deadline from the job's JobSuspended=True transition time.

• If the cluster recovers before the deadline, the operator resumes the job and the backup continues
• If the deadline expires, the operator marks the backup as Failed, deletes the suspended job, and releases its backup lease.

CHECKLIST

Jira

• Is the Jira ticket created and referenced properly?
• Does the Jira ticket have the proper statuses for documentation (Needs Doc) and QA (Needs QA)?
• Does the Jira ticket link to the proper milestone (Fix Version field)?

Tests

• Is an E2E test/test case added for the new feature/change?
• Are unit tests added where appropriate?

Config/Logging/Testability

• Are all needed new/changed options added to default YAML files?
• Are all needed new/changed options added to the Helm Chart?
• Did we add proper logging messages for operator actions?
• Did we ensure compatibility with the previous version or cluster upgrade process?
• Does the change support oldest and newest supported PS version?
• Does the change support oldest and newest supported Kubernetes version?

[JNKPercona]

Test Name	Result	Time
async-ignore-annotations-8-4	passed	00:06:39
async-global-metadata-8-4	passed	00:15:21
async-upgrade-8-0	passed	00:13:20
async-upgrade-8-4	passed	00:13:51
auto-config-8-4	passed	00:18:00
config-8-4	passed	00:27:48
config-router-8-0	passed	00:07:12
config-router-8-4	passed	00:07:38
custom-users-8-4	passed	00:05:57
demand-backup-8-0	failure	00:08:22
demand-backup-8-4	failure	00:14:13
gr-pitr-minio-8-4	passed	00:26:51
gr-pitr-encrypted-minio-8-4	passed	00:21:13
gr-pitr-one-pod-8-4	passed	00:12:14
async-pitr-minio-8-4	passed	00:25:29
demand-backup-cloud-8-4	passed	00:23:10
demand-backup-retry-8-4	failure	00:08:32
demand-backup-incremental-8-0	passed	00:35:27
demand-backup-incremental-8-4	passed	00:36:44
async-data-at-rest-encryption-8-0	passed	00:14:00
async-data-at-rest-encryption-8-4	passed	00:16:02
gr-cross-cluster-8-0	passed	00:20:48
gr-cross-cluster-8-4	passed	00:20:37
gr-cross-cluster-backup-8-0	passed	00:25:12
gr-cross-cluster-backup-8-4	passed	00:26:39
gr-global-metadata-8-4	passed	00:13:28
gr-data-at-rest-encryption-8-0	passed	00:16:13
gr-data-at-rest-encryption-8-4	passed	00:14:48
gr-demand-backup-8-4	passed	00:13:59
gr-demand-backup-cloud-8-4	passed	00:22:35
gr-demand-backup-haproxy-8-4	passed	00:10:43
gr-demand-backup-incremental-8-0	passed	00:25:12
gr-demand-backup-incremental-8-4	passed	00:24:07
gr-demand-backup-incremental-compressed-8-0	passed	00:11:19
gr-demand-backup-incremental-compressed-8-4	passed	00:11:45
gr-demand-backup-incremental-encrypted-8-0	passed	00:18:16
gr-demand-backup-incremental-encrypted-8-4	passed	00:18:28
gr-finalizer-8-4	passed	00:06:05
gr-haproxy-8-0	passed	00:04:59
gr-haproxy-8-4	passed	00:04:07
gr-ignore-annotations-8-4	passed	00:04:51
gr-init-deploy-8-0	passed	00:11:43
gr-init-deploy-8-4	passed	00:09:32
gr-one-pod-8-4	passed	00:05:32
gr-recreate-8-4	passed	00:17:42
gr-scaling-8-4	passed	00:07:55
gr-scheduled-backup-8-4	passed	00:27:03
gr-scheduled-backup-incremental-8-4	passed	00:35:55
gr-security-context-8-4	passed	00:09:24
gr-self-healing-8-4	failure	00:34:36
gr-tls-cert-manager-8-4	passed	00:11:29
gr-users-8-4	passed	00:07:43
gr-upgrade-8-0	passed	00:10:33
gr-upgrade-8-4	passed	00:10:12
haproxy-8-0	passed	00:08:34
haproxy-8-4	passed	00:08:44
init-deploy-8-0	passed	00:07:10
init-deploy-8-4	passed	00:06:01
limits-8-4	passed	00:04:26
monitoring-8-4	passed	00:19:32
one-pod-8-0	passed	00:06:23
one-pod-8-4	passed	00:06:03
operator-self-healing-8-4	passed	00:11:44
pvc-auto-resize-8-4	passed	00:04:50
pvc-resize-8-4	passed	00:14:03
recreate-8-4	passed	00:12:45
scaling-8-4	passed	00:10:44
scheduled-backup-8-0	failure	00:31:49
scheduled-backup-8-4	passed	00:26:35
scheduled-backup-incremental-8-0	passed	00:34:12
scheduled-backup-incremental-8-4	passed	00:35:11
service-per-pod-8-4	passed	00:07:03
sidecars-8-4	passed	00:04:53
smart-update-8-4	passed	00:09:41
storage-8-4	passed	00:04:15
switch-cluster-type-8-4	passed	00:11:03
telemetry-8-4	passed	00:06:26
tls-cert-manager-8-4	passed	00:16:54
users-8-0	passed	00:07:42
users-8-4	passed	00:08:05
version-service-8-4	passed	00:20:29

Summary	Value
Tests Run	81/81
Job Duration	03:10:43
Total Test Time	20:17:28

commit: 68b15ba
image: perconalab/percona-server-mysql-operator:PR-1528-68b15bac